2024-04-15T15:04:06.948 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T15:04:06.948 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} --verbose --archive /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121 --name leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/orig.config.yaml 2024-04-15T15:04:07.099 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:04:07.205 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121 branch: wip-lusov-quiesce-xlock description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} email: null first_in_suite: false job_id: '7657121' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiesce-xlock ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiesce-xlock sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 2288 sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 sleep_before_teardown: 0 suite: fs suite_branch: wip-lusov-quiesce-xlock suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoB5nFh91vNFMaqMdEZvAO3Q5iV1zSQy++mNWxmIrkasO1wNgVOeZSJqySUWA+WXmqhQI2uCfETN2LpmTmCYXQ= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJEsMQ1pqCEDgo13JQ7ZcpjjN11GlQS6KwaEcdrg7vuceBTRN9SO5GlKQAlCVhLwlf/pgcNfejnvWruRhTvlfgA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_quiesce teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_14:46:31 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T15:04:07.206 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa; will attempt to use it 2024-04-15T15:04:07.207 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks 2024-04-15T15:04:07.207 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T15:04:07.217 INFO:teuthology.task.internal:Checking packages... 2024-04-15T15:04:07.243 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14' 2024-04-15T15:04:07.243 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T15:04:07.243 INFO:teuthology.packaging:ref: None 2024-04-15T15:04:07.243 INFO:teuthology.packaging:tag: None 2024-04-15T15:04:07.243 INFO:teuthology.packaging:branch: wip-lusov-quiesce-xlock 2024-04-15T15:04:07.243 INFO:teuthology.packaging:sha1: 9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:04:07.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiesce-xlock 2024-04-15T15:04:07.394 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2861.g9bf1c4ba 2024-04-15T15:04:07.396 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T15:04:07.596 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T15:04:07.596 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T15:04:07.647 INFO:teuthology.task.internal:Saving configuration 2024-04-15T15:04:07.667 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T15:04:07.673 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:04:07.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:07.741710', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoB5nFh91vNFMaqMdEZvAO3Q5iV1zSQy++mNWxmIrkasO1wNgVOeZSJqySUWA+WXmqhQI2uCfETN2LpmTmCYXQ='} 2024-04-15T15:04:07.725 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:07.739664', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJEsMQ1pqCEDgo13JQ7ZcpjjN11GlQS6KwaEcdrg7vuceBTRN9SO5GlKQAlCVhLwlf/pgcNfejnvWruRhTvlfgA='} 2024-04-15T15:04:07.725 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T15:04:07.733 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-15T15:04:07.733 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-15T15:04:07.733 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T15:04:07.809 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f57064580d0>, signals=[15]) 2024-04-15T15:04:07.810 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T15:04:07.817 INFO:teuthology.task.internal:Opening connections... 2024-04-15T15:04:07.817 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:04:07.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:04:07.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:04:07.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:04:07.999 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T15:04:08.005 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-04-15T15:04:08.023 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-04-15T15:04:08.023 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-04-15T15:04:08.084 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-04-15T15:04:08.084 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-04-15T15:04:08.084 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-04-15T15:04:08.084 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:04:08.084 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:04:08.085 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:04:08.086 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-04-15T15:04:08.112 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-04-15T15:04:08.129 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-04-15T15:04:08.129 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-04-15T15:04:08.186 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-04-15T15:04:08.186 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-04-15T15:04:08.186 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T15:04:08.187 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T15:04:08.187 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-04-15T15:04:08.212 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T15:04:08.221 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T15:04:08.237 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T15:04:08.237 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:04:08.241 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-04-15T15:04:08.257 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T15:04:08.270 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T15:04:08.270 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:04:08.298 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-04-15T15:04:08.397 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T15:04:08.459 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T15:04:08.476 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-15T15:04:08.476 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-15T15:04:08.476 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:04:08.477 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T15:04:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:04:08.494 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-15T15:04:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:04:08.496 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-04-15T15:04:08.550 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:04:08.551 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-361.el9.x86_64 2024-04-15T15:04:08.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-04-15T15:04:08.553 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-361.el9.x86_64 2024-04-15T15:04:08.553 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-361.el9.x86_64 2024-04-15T15:04:08.553 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-04-15T15:04:10.674 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-15T15:04:10.728 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-15T15:04:12.082 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 6.7 MB/s | 2.0 MB 00:00 2024-04-15T15:04:12.086 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-04-15T15:04:13.885 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 5.5 MB/s | 8.0 MB 00:01 2024-04-15T15:04:14.139 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-04-15T15:04:16.902 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 9.8 MB/s | 19 MB 00:01 2024-04-15T15:04:17.506 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 19 MB 00:02 2024-04-15T15:04:21.740 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-04-15T15:04:23.383 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-15T15:04:24.969 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-15T15:04:29.485 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 696 kB/s | 6.2 MB 00:09 2024-04-15T15:04:30.008 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-04-15T15:04:31.135 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-15T15:04:32.365 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:04:32.365 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:04:32.455 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:04:32.467 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 77 M 2024-04-15T15:04:32.468 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 123 M 2024-04-15T15:04:32.469 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-15T15:04:32.751 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 983 kB/s | 277 kB 00:00 2024-04-15T15:04:33.534 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 14 MB/s | 15 MB 00:01 2024-04-15T15:04:34.176 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-core-5.14.0-438.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-15T15:04:34.501 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 17 MB/s | 34 MB 00:02 2024-04-15T15:04:34.504 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:04:34.505 INFO:teuthology.orchestra.run.smithi096.stdout:Total 38 MB/s | 77 MB 00:02 2024-04-15T15:04:34.505 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-15T15:04:34.887 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-15T15:04:34.887 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-15T15:04:35.730 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-15T15:04:35.731 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-15T15:04:38.038 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-15T15:04:38.664 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:04:38.753 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:41.518 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:04:41.662 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:46.350 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:04:46.386 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:51.170 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:04:54.977 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 899 kB/s | 21 MB 00:23 2024-04-15T15:04:59.931 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-15T15:05:02.296 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T15:05:02.296 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T15:05:02.402 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-15T15:05:02.406 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-15T15:05:02.406 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-04-15T15:05:02.407 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-04-15T15:05:02.408 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:02.408 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 77 M 2024-04-15T15:05:02.408 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 123 M 2024-04-15T15:05:02.408 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-15T15:05:02.574 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.6 MB/s | 277 kB 00:00 2024-04-15T15:05:03.066 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-04-15T15:05:03.666 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-04-15T15:05:04.208 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-15T15:05:04.211 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:05:04.211 INFO:teuthology.orchestra.run.smithi070.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-15T15:05:04.211 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-15T15:05:04.588 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-15T15:05:04.588 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-15T15:05:05.427 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-15T15:05:05.427 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-15T15:05:07.747 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-15T15:05:08.354 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:08.441 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:11.196 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:11.326 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:16.023 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:16.151 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:20.674 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:20.675 INFO:teuthology.orchestra.run.smithi096.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:05:20.675 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:05:20.718 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:20.918 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:21.115 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:21.115 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:21.115 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:21.115 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:05:21.552 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-15T15:05:21.875 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:21.989 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:21.989 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:21.990 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:05:21.990 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:05:21.990 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:05:21.990 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T15:05:21.990 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:05:21.990 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:22.070 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:22.070 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:22.071 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 68.el9 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8264254 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:05:22.105 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-04-15T15:05:22.106 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:05:22.107 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:05:22.107 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:05:22.107 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:05:22.108 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:05:22.723 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-04-15T15:05:24.165 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:24.224 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-04-15T15:05:24.226 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-04-15T15:05:24.241 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.059OCuNU27 2024-04-15T15:05:24.242 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.059OCuNU27 2024-04-15T15:05:24.310 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.059OCuNU27 2024-04-15T15:05:24.452 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.059OCuNU27 is 6KB 2024-04-15T15:05:24.500 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.059OCuNU27 2024-04-15T15:05:24.514 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:24.579 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:24.579 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:24.580 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:24.580 INFO:teuthology.orchestra.run.smithi096.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:24.581 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:24.739 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-04-15T15:05:50.182 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:50.183 INFO:teuthology.orchestra.run.smithi070.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T15:05:50.183 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:50.222 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:50.569 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:50.569 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T15:05:50.569 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T15:05:50.569 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:50.993 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-15T15:05:51.309 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:51.451 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:51.452 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:51.452 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T15:05:51.452 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T15:05:51.453 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T15:05:51.453 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T15:05:51.453 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T15:05:51.453 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T15:05:51.533 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T15:05:51.533 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T15:05:51.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-04-15T15:05:51.576 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-04-15T15:05:51.576 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-04-15T15:05:51.576 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-04-15T15:05:51.576 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 68.el9 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8264254 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-04-15T15:05:51.577 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:05:51.578 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T15:05:51.579 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T15:05:51.579 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-438.el9.x86_64 2024-04-15T15:05:51.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T15:05:52.228 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-04-15T15:05:53.630 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T15:05:53.665 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-04-15T15:05:53.667 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-15T15:05:53.683 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.L0iPLf5wrz 2024-04-15T15:05:53.683 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.L0iPLf5wrz 2024-04-15T15:05:53.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.L0iPLf5wrz 2024-04-15T15:05:53.892 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.L0iPLf5wrz is 6KB 2024-04-15T15:05:53.941 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.L0iPLf5wrz 2024-04-15T15:05:53.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T15:05:54.027 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T15:05:54.027 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T15:05:54.027 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T15:05:54.027 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T15:05:54.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T15:05:54.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-04-15T15:05:54.743 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T15:05:54.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:05:54.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:13.155 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-04-15T15:06:22.162 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:06:22.163 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:24.185 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T15:06:24.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:06:24.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:25.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-04-15T15:06:37.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:06:37.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:06:42.593 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-04-15T15:06:51.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:06:51.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:07:09.037 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-04-15T15:07:09.413 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:07:09.413 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:07:09.413 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-04-15T15:07:09.428 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:07:09.429 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:07:09.429 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:07:09.429 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:07:10.429 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T15:07:10.430 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:07:10.530 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-04-15T15:07:10.565 DEBUG:teuthology.parallel:result is None 2024-04-15T15:07:51.607 DEBUG:teuthology.orchestra.remote:timed out 2024-04-15T15:08:03.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:08:03.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:08:03.997 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-15T15:08:04.339 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:08:04.339 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T15:08:04.339 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-04-15T15:08:04.355 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-438.el9.x86_64 2024-04-15T15:08:04.356 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T15:08:04.356 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T15:08:04.356 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T15:08:05.357 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T15:08:05.358 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T15:08:05.420 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-04-15T15:08:05.462 DEBUG:teuthology.parallel:result is None 2024-04-15T15:08:05.462 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T15:08:05.469 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T15:08:05.470 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:08:05.472 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T15:08:05.492 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T15:08:05.632 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T15:08:05.720 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T15:08:05.720 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:08:05.723 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T15:08:05.756 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T15:08:05.762 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T15:08:05.762 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:08:05.786 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T15:08:05.825 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:05.830 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:05.840 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:05.845 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T15:08:05.847 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T15:08:05.854 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T15:08:05.854 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:08:05.884 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T15:08:05.918 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T15:08:05.928 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T15:08:05.928 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:08:05.958 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T15:08:05.981 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:06.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:06.118 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:06.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:06.249 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:08:06.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:08:06.325 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:06.378 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T15:08:06.434 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:06.517 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T15:08:06.569 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:08:06.569 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T15:08:06.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-04-15T15:08:06.640 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-04-15T15:08:06.693 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:08:06.742 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:08:07.178 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T15:08:07.186 INFO:teuthology.task.internal:Starting timer... 2024-04-15T15:08:07.186 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T15:08:07.200 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T15:08:07.210 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-15T15:08:07.211 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-04-15T15:08:07.280 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-04-15T15:08:07.281 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-04-15T15:08:07.353 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-04-15T15:08:07.355 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T15:08:07.355 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-04-15T15:08:07.377 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-04-15T15:08:07.377 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-04-15T15:08:07.449 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-04-15T15:08:07.450 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive'} 2024-04-15T15:08:07.450 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:08:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:08:07.480 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:08:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:08:07.511 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T15:08:07.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:08:07.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-04-15T15:08:07.610 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T15:08:07.619 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T15:08:07.635 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T15:08:07.636 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi070.front.sepia.ceph.com,smithi096.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T15:12:22.339 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com')] 2024-04-15T15:12:22.341 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:12:22.342 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:22.433 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-04-15T15:12:22.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-04-15T15:12:22.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:12:22.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T15:12:22.611 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-04-15T15:12:22.698 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-04-15T15:12:22.698 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T15:12:22.711 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T15:12:22.711 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:12:22.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:12:22.715 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T15:12:22.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:12:22.745 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:12:22.765 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:12:22.771 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T15:12:22.789 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T15:12:22.797 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-04-15T15:12:22.811 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-04-15T15:12:22.820 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-04-15T15:12:22.829 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:12:22.835 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-04-15T15:12:22.846 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:12:22.852 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T15:12:22.870 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T15:12:22.906 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.909 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.927 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:12:22.931 INFO:teuthology.run_tasks:Running task install... 2024-04-15T15:12:22.940 DEBUG:teuthology.task.install:project ceph 2024-04-15T15:12:22.940 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}} 2024-04-15T15:12:22.940 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'} 2024-04-15T15:12:22.940 INFO:teuthology.task.install:Using flavor: default 2024-04-15T15:12:22.947 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-15T15:12:22.947 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-15T15:12:22.948 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:12:22.948 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:12:22.951 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'tag': None, 'wait_for_package': False} 2024-04-15T15:12:22.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:12:23.096 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:12:23.098 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:12:23.103 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/ 2024-04-15T15:12:23.103 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2861.g9bf1c4ba 2024-04-15T15:12:23.213 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:12:23.213 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:12:23.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:12:23.243 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:12:23.243 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:12:23.271 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-lusov-quiesce-xlock/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-15T15:12:23.271 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:12:23.274 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-15T15:12:23.304 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-15T15:12:23.304 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-15T15:12:23.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:12:23.373 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-15T15:12:23.397 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:12:23.424 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-04-15T15:12:23.426 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-04-15T15:12:23.457 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-15T15:12:23.524 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-04-15T15:12:23.526 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-04-15T15:12:23.670 INFO:teuthology.orchestra.run.smithi070.stdout:74 files removed 2024-04-15T15:12:23.697 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:12:23.769 INFO:teuthology.orchestra.run.smithi096.stdout:74 files removed 2024-04-15T15:12:23.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-15T15:12:24.426 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 196 kB/s | 83 kB 00:00 2024-04-15T15:12:24.611 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 162 kB/s | 83 kB 00:00 2024-04-15T15:12:25.060 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 20 kB/s | 11 kB 00:00 2024-04-15T15:12:25.078 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-15T15:12:25.427 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-04-15T15:12:25.445 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-04-15T15:12:25.938 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-04-15T15:12:26.481 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.7 MB 00:00 2024-04-15T15:12:27.379 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-04-15T15:12:27.915 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-04-15T15:12:28.789 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-15T15:12:29.080 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 5.8 MB/s | 8.0 MB 00:01 2024-04-15T15:12:30.939 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-15T15:12:31.432 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-04-15T15:12:34.974 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-15T15:12:36.824 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-04-15T15:12:37.007 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-15T15:12:38.715 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-04-15T15:12:39.569 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-04-15T15:12:40.783 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-04-15T15:12:44.575 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-15T15:12:45.800 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-15T15:12:47.262 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:47.263 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:47.277 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:12:47.278 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:12:47.279 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:12:47.280 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:47.282 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:47.285 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:12:47.286 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:12:47.365 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:12:47.378 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:12:47.379 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:12:47.380 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:12:47.381 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:12:47.382 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:12:47.383 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:12:47.383 INFO:teuthology.orchestra.run.smithi070.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:12:47.383 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:12:47.384 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:12:47.385 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:12:47.386 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:12:47.387 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:12:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:12:47.389 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:12:47.390 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:12:47.391 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:12:47.392 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:12:47.393 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout:Install 150 Packages 2024-04-15T15:12:47.394 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 24 Packages 2024-04-15T15:12:47.395 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:12:47.395 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 228 M 2024-04-15T15:12:47.395 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-04-15T15:12:48.034 INFO:teuthology.orchestra.run.smithi070.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-15T15:12:48.209 INFO:teuthology.orchestra.run.smithi070.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 5.0 MB/s | 890 kB 00:00 2024-04-15T15:12:48.259 INFO:teuthology.orchestra.run.smithi070.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 2.8 MB/s | 145 kB 00:00 2024-04-15T15:12:48.484 INFO:teuthology.orchestra.run.smithi070.stdout:(4/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 10 MB/s | 2.4 MB 00:00 2024-04-15T15:12:48.497 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:48.498 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-15T15:12:48.511 INFO:teuthology.orchestra.run.smithi096.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-04-15T15:12:48.512 INFO:teuthology.orchestra.run.smithi096.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-15T15:12:48.514 INFO:teuthology.orchestra.run.smithi096.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-04-15T15:12:48.515 INFO:teuthology.orchestra.run.smithi096.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:48.517 INFO:teuthology.orchestra.run.smithi096.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-15T15:12:48.519 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-04-15T15:12:48.521 INFO:teuthology.orchestra.run.smithi096.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-04-15T15:12:48.601 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 6.6 k 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.4 M 2024-04-15T15:12:48.628 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 890 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 145 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 1.5 M 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 142 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 3.5 M 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 7.4 M 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 49 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 10 M 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 49 M 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 264 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 762 k 2024-04-15T15:12:48.629 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 224 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 27 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 18 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 32 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 747 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 127 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-15T15:12:48.630 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql x86_64 13.14-1.el9 CentOS-AppStream 1.5 M 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib x86_64 13.14-1.el9 CentOS-AppStream 817 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server x86_64 13.14-1.el9 CentOS-AppStream 5.7 M 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 162 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 320 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 297 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 99 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 85 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 171 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-15T15:12:48.631 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager x86_64 1:1.47.4-1.el9 CentOS-BaseOS 2.3 M 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm x86_64 1:1.47.4-1.el9 CentOS-BaseOS 1.8 M 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team x86_64 1:1.47.4-1.el9 CentOS-BaseOS 41 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui x86_64 1:1.47.4-1.el9 CentOS-BaseOS 248 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-04-15T15:12:48.632 INFO:teuthology.orchestra.run.smithi096.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.4 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 3.1 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-04-15T15:12:48.633 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 21 M 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 34 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 2.4 M 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 249 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 4.7 M 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 17 M 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2861.g9bf1c4ba.el8 ceph-noarch 16 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 25 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-04-15T15:12:48.634 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 164 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 501 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-04-15T15:12:48.635 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 5.2 M 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-04-15T15:12:48.636 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs x86_64 13.14-1.el9 CentOS-AppStream 132 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 45 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.0.0-2861.g9bf1c4ba.el8 ceph 135 k 2024-04-15T15:12:48.637 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-15T15:12:48.638 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-04-15T15:12:48.639 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-15T15:12:48.640 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-04-15T15:12:48.641 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-04-15T15:12:48.642 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout:============================================================================================ 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout:Install 150 Packages 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 24 Packages 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:12:48.643 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 228 M 2024-04-15T15:12:48.644 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-04-15T15:12:48.659 INFO:teuthology.orchestra.run.smithi070.stdout:(5/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 6.7 MB/s | 5.4 MB 00:00 2024-04-15T15:12:48.693 INFO:teuthology.orchestra.run.smithi070.stdout:(6/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 7.2 MB/s | 1.5 MB 00:00 2024-04-15T15:12:49.060 INFO:teuthology.orchestra.run.smithi070.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 12 MB/s | 4.7 MB 00:00 2024-04-15T15:12:49.651 INFO:teuthology.orchestra.run.smithi096.stdout:(1/174): ceph-19.0.0-2861.g9bf1c4ba.el8.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-15T15:12:49.852 INFO:teuthology.orchestra.run.smithi070.stdout:(8/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el8 13 MB/s | 10 MB 00:00 2024-04-15T15:12:49.952 INFO:teuthology.orchestra.run.smithi070.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 252 kB/s | 25 kB 00:00 2024-04-15T15:12:50.026 INFO:teuthology.orchestra.run.smithi096.stdout:(2/174): ceph-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 2.3 MB/s | 890 kB 00:00 2024-04-15T15:12:50.126 INFO:teuthology.orchestra.run.smithi096.stdout:(3/174): ceph-immutable-object-cache-19.0.0-286 1.4 MB/s | 145 kB 00:00 2024-04-15T15:12:50.303 INFO:teuthology.orchestra.run.smithi070.stdout:(10/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 10 MB/s | 17 MB 00:01 2024-04-15T15:12:50.386 INFO:teuthology.orchestra.run.smithi070.stdout:(11/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 2.6 MB/s | 224 kB 00:00 2024-04-15T15:12:50.470 INFO:teuthology.orchestra.run.smithi070.stdout:(12/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 324 kB/s | 27 kB 00:00 2024-04-15T15:12:50.545 INFO:teuthology.orchestra.run.smithi070.stdout:(13/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 424 kB/s | 32 kB 00:00 2024-04-15T15:12:50.551 INFO:teuthology.orchestra.run.smithi096.stdout:(4/174): ceph-mds-19.0.0-2861.g9bf1c4ba.el8.x86 5.6 MB/s | 2.4 MB 00:00 2024-04-15T15:12:50.679 INFO:teuthology.orchestra.run.smithi070.stdout:(14/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 5.5 MB/s | 747 kB 00:00 2024-04-15T15:12:50.771 INFO:teuthology.orchestra.run.smithi070.stdout:(15/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.7 MB/s | 164 kB 00:00 2024-04-15T15:12:50.837 INFO:teuthology.orchestra.run.smithi070.stdout:(16/174): librados-devel-19.0.0-2861.g9bf1c4ba. 1.9 MB/s | 127 kB 00:00 2024-04-15T15:12:50.851 INFO:teuthology.orchestra.run.smithi096.stdout:(5/174): ceph-mgr-19.0.0-2861.g9bf1c4ba.el8.x86 5.0 MB/s | 1.5 MB 00:00 2024-04-15T15:12:50.921 INFO:teuthology.orchestra.run.smithi070.stdout:(17/174): libradosstriper1-19.0.0-2861.g9bf1c4b 5.9 MB/s | 501 kB 00:00 2024-04-15T15:12:51.635 INFO:teuthology.orchestra.run.smithi096.stdout:(6/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8. 9.1 MB/s | 21 MB 00:02 2024-04-15T15:12:51.705 INFO:teuthology.orchestra.run.smithi070.stdout:(18/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 6.7 MB/s | 5.2 MB 00:00 2024-04-15T15:12:51.727 INFO:teuthology.orchestra.run.smithi096.stdout:(7/174): ceph-mon-19.0.0-2861.g9bf1c4ba.el8.x86 5.4 MB/s | 4.7 MB 00:00 2024-04-15T15:12:51.771 INFO:teuthology.orchestra.run.smithi070.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 680 kB/s | 45 kB 00:00 2024-04-15T15:12:51.855 INFO:teuthology.orchestra.run.smithi070.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.6 MB/s | 135 kB 00:00 2024-04-15T15:12:51.939 INFO:teuthology.orchestra.run.smithi070.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.9 MB/s | 162 kB 00:00 2024-04-15T15:12:52.039 INFO:teuthology.orchestra.run.smithi070.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 3.1 MB/s | 320 kB 00:00 2024-04-15T15:12:52.060 INFO:teuthology.orchestra.run.smithi096.stdout:(8/174): ceph-base-19.0.0-2861.g9bf1c4ba.el8.x8 2.0 MB/s | 5.4 MB 00:02 2024-04-15T15:12:52.106 INFO:teuthology.orchestra.run.smithi070.stdout:(23/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 4.4 MB/s | 297 kB 00:00 2024-04-15T15:12:52.154 INFO:teuthology.orchestra.run.smithi096.stdout:(9/174): ceph-selinux-19.0.0-2861.g9bf1c4ba.el8 275 kB/s | 25 kB 00:00 2024-04-15T15:12:52.189 INFO:teuthology.orchestra.run.smithi070.stdout:(24/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 1.2 MB/s | 99 kB 00:00 2024-04-15T15:12:52.281 INFO:teuthology.orchestra.run.smithi070.stdout:(25/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 931 kB/s | 85 kB 00:00 2024-04-15T15:12:52.923 INFO:teuthology.orchestra.run.smithi070.stdout:(26/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 4.8 MB/s | 3.1 MB 00:00 2024-04-15T15:12:53.015 INFO:teuthology.orchestra.run.smithi070.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 1.8 MB/s | 171 kB 00:00 2024-04-15T15:12:53.099 INFO:teuthology.orchestra.run.smithi070.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 411 kB/s | 34 kB 00:00 2024-04-15T15:12:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 1.5 MB/s | 142 kB 00:00 2024-04-15T15:12:53.386 INFO:teuthology.orchestra.run.smithi096.stdout:(10/174): ceph-radosgw-19.0.0-2861.g9bf1c4ba.el 6.0 MB/s | 10 MB 00:01 2024-04-15T15:12:53.561 INFO:teuthology.orchestra.run.smithi096.stdout:(11/174): ceph-osd-19.0.0-2861.g9bf1c4ba.el8.x8 8.7 MB/s | 17 MB 00:01 2024-04-15T15:12:53.586 INFO:teuthology.orchestra.run.smithi096.stdout:(12/174): cephfs-mirror-19.0.0-2861.g9bf1c4ba.e 1.1 MB/s | 224 kB 00:00 2024-04-15T15:12:53.620 INFO:teuthology.orchestra.run.smithi096.stdout:(13/174): cephfs-shell-19.0.0-2861.g9bf1c4ba.el 464 kB/s | 27 kB 00:00 2024-04-15T15:12:53.645 INFO:teuthology.orchestra.run.smithi096.stdout:(14/174): libcephfs-devel-19.0.0-2861.g9bf1c4ba 543 kB/s | 32 kB 00:00 2024-04-15T15:12:53.674 INFO:teuthology.orchestra.run.smithi070.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 7.3 MB/s | 3.5 MB 00:00 2024-04-15T15:12:53.720 INFO:teuthology.orchestra.run.smithi096.stdout:(15/174): libcephfs2-19.0.0-2861.g9bf1c4ba.el8. 7.3 MB/s | 747 kB 00:00 2024-04-15T15:12:53.746 INFO:teuthology.orchestra.run.smithi096.stdout:(16/174): libcephsqlite-19.0.0-2861.g9bf1c4ba.e 1.6 MB/s | 164 kB 00:00 2024-04-15T15:12:53.779 INFO:teuthology.orchestra.run.smithi096.stdout:(17/174): librados-devel-19.0.0-2861.g9bf1c4ba. 2.1 MB/s | 127 kB 00:00 2024-04-15T15:12:53.888 INFO:teuthology.orchestra.run.smithi096.stdout:(18/174): libradosstriper1-19.0.0-2861.g9bf1c4b 3.4 MB/s | 501 kB 00:00 2024-04-15T15:12:53.971 INFO:teuthology.orchestra.run.smithi096.stdout:(19/174): python3-ceph-argparse-19.0.0-2861.g9b 542 kB/s | 45 kB 00:00 2024-04-15T15:12:54.063 INFO:teuthology.orchestra.run.smithi096.stdout:(20/174): python3-ceph-common-19.0.0-2861.g9bf1 1.4 MB/s | 135 kB 00:00 2024-04-15T15:12:54.163 INFO:teuthology.orchestra.run.smithi096.stdout:(21/174): python3-cephfs-19.0.0-2861.g9bf1c4ba. 1.6 MB/s | 162 kB 00:00 2024-04-15T15:12:54.264 INFO:teuthology.orchestra.run.smithi096.stdout:(22/174): python3-rados-19.0.0-2861.g9bf1c4ba.e 3.1 MB/s | 320 kB 00:00 2024-04-15T15:12:54.364 INFO:teuthology.orchestra.run.smithi096.stdout:(23/174): python3-rbd-19.0.0-2861.g9bf1c4ba.el8 2.9 MB/s | 297 kB 00:00 2024-04-15T15:12:54.439 INFO:teuthology.orchestra.run.smithi096.stdout:(24/174): librgw2-19.0.0-2861.g9bf1c4ba.el8.x86 7.9 MB/s | 5.2 MB 00:00 2024-04-15T15:12:54.465 INFO:teuthology.orchestra.run.smithi096.stdout:(25/174): python3-rgw-19.0.0-2861.g9bf1c4ba.el8 987 kB/s | 99 kB 00:00 2024-04-15T15:12:54.507 INFO:teuthology.orchestra.run.smithi096.stdout:(26/174): rbd-fuse-19.0.0-2861.g9bf1c4ba.el8.x8 1.2 MB/s | 85 kB 00:00 2024-04-15T15:12:54.590 INFO:teuthology.orchestra.run.smithi096.stdout:(27/174): rbd-nbd-19.0.0-2861.g9bf1c4ba.el8.x86 2.0 MB/s | 171 kB 00:00 2024-04-15T15:12:54.682 INFO:teuthology.orchestra.run.smithi096.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2861.g 373 kB/s | 34 kB 00:00 2024-04-15T15:12:54.766 INFO:teuthology.orchestra.run.smithi096.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2861.g9bf1c4b 1.7 MB/s | 142 kB 00:00 2024-04-15T15:12:55.008 INFO:teuthology.orchestra.run.smithi096.stdout:(30/174): rbd-mirror-19.0.0-2861.g9bf1c4ba.el8. 5.7 MB/s | 3.1 MB 00:00 2024-04-15T15:12:55.033 INFO:teuthology.orchestra.run.smithi070.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 5.4 MB/s | 7.4 MB 00:01 2024-04-15T15:12:55.108 INFO:teuthology.orchestra.run.smithi070.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 3.3 MB/s | 249 kB 00:00 2024-04-15T15:12:55.183 INFO:teuthology.orchestra.run.smithi070.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 660 kB/s | 49 kB 00:00 2024-04-15T15:12:55.200 INFO:teuthology.orchestra.run.smithi096.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2861.g9bf1c 8.2 MB/s | 3.5 MB 00:00 2024-04-15T15:12:55.226 INFO:teuthology.orchestra.run.smithi070.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 396 kB/s | 16 kB 00:00 2024-04-15T15:12:55.267 INFO:teuthology.orchestra.run.smithi096.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2861.g9b 3.7 MB/s | 249 kB 00:00 2024-04-15T15:12:55.317 INFO:teuthology.orchestra.run.smithi070.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 2.8 MB/s | 264 kB 00:00 2024-04-15T15:12:55.325 INFO:teuthology.orchestra.run.smithi096.stdout:(33/174): ceph-mgr-rook-19.0.0-2861.g9bf1c4ba.e 851 kB/s | 49 kB 00:00 2024-04-15T15:12:55.392 INFO:teuthology.orchestra.run.smithi096.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2861.g9 246 kB/s | 16 kB 00:00 2024-04-15T15:12:55.450 INFO:teuthology.orchestra.run.smithi070.stdout:(36/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 5.6 MB/s | 762 kB 00:00 2024-04-15T15:12:55.467 INFO:teuthology.orchestra.run.smithi096.stdout:(35/174): ceph-volume-19.0.0-2861.g9bf1c4ba.el8 3.4 MB/s | 264 kB 00:00 2024-04-15T15:12:55.509 INFO:teuthology.orchestra.run.smithi070.stdout:(37/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 312 kB/s | 18 kB 00:00 2024-04-15T15:12:55.584 INFO:teuthology.orchestra.run.smithi096.stdout:(36/174): cephadm-19.0.0-2861.g9bf1c4ba.el8.noa 6.4 MB/s | 762 kB 00:00 2024-04-15T15:12:55.643 INFO:teuthology.orchestra.run.smithi096.stdout:(37/174): cephfs-top-19.0.0-2861.g9bf1c4ba.el8. 314 kB/s | 18 kB 00:00 2024-04-15T15:12:55.651 INFO:teuthology.orchestra.run.smithi070.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.6 MB/s | 662 kB 00:00 2024-04-15T15:12:55.710 INFO:teuthology.orchestra.run.smithi070.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 15 MB/s | 920 kB 00:00 2024-04-15T15:12:55.744 INFO:teuthology.orchestra.run.smithi070.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-04-15T15:12:55.776 INFO:teuthology.orchestra.run.smithi096.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 4.9 MB/s | 662 kB 00:00 2024-04-15T15:12:55.785 INFO:teuthology.orchestra.run.smithi070.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-04-15T15:12:55.811 INFO:teuthology.orchestra.run.smithi070.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-04-15T15:12:55.827 INFO:teuthology.orchestra.run.smithi096.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 18 MB/s | 920 kB 00:00 2024-04-15T15:12:55.877 INFO:teuthology.orchestra.run.smithi096.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-04-15T15:12:55.903 INFO:teuthology.orchestra.run.smithi070.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-04-15T15:12:55.919 INFO:teuthology.orchestra.run.smithi096.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-04-15T15:12:55.937 INFO:teuthology.orchestra.run.smithi070.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 514 kB/s | 17 kB 00:00 2024-04-15T15:12:55.962 INFO:teuthology.orchestra.run.smithi070.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 406 kB/s | 10 kB 00:00 2024-04-15T15:12:56.004 INFO:teuthology.orchestra.run.smithi070.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-04-15T15:12:56.011 INFO:teuthology.orchestra.run.smithi096.stdout:(42/174): ceph-mgr-diskprediction-local-19.0.0- 7.4 MB/s | 7.4 MB 00:01 2024-04-15T15:12:56.029 INFO:teuthology.orchestra.run.smithi070.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-04-15T15:12:56.036 INFO:teuthology.orchestra.run.smithi096.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 271 kB/s | 32 kB 00:00 2024-04-15T15:12:56.096 INFO:teuthology.orchestra.run.smithi070.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-04-15T15:12:56.120 INFO:teuthology.orchestra.run.smithi096.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-04-15T15:12:56.137 INFO:teuthology.orchestra.run.smithi070.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-04-15T15:12:56.145 INFO:teuthology.orchestra.run.smithi096.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 160 kB/s | 17 kB 00:00 2024-04-15T15:12:56.163 INFO:teuthology.orchestra.run.smithi070.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-04-15T15:12:56.170 INFO:teuthology.orchestra.run.smithi096.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 201 kB/s | 10 kB 00:00 2024-04-15T15:12:56.195 INFO:teuthology.orchestra.run.smithi096.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-04-15T15:12:56.205 INFO:teuthology.orchestra.run.smithi070.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-04-15T15:12:56.221 INFO:teuthology.orchestra.run.smithi096.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-04-15T15:12:56.230 INFO:teuthology.orchestra.run.smithi070.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2024-04-15T15:12:56.255 INFO:teuthology.orchestra.run.smithi096.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-04-15T15:12:56.264 INFO:teuthology.orchestra.run.smithi070.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 6.4 MB/s | 217 kB 00:00 2024-04-15T15:12:56.280 INFO:teuthology.orchestra.run.smithi096.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-04-15T15:12:56.297 INFO:teuthology.orchestra.run.smithi070.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-04-15T15:12:56.305 INFO:teuthology.orchestra.run.smithi096.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 945 kB/s | 46 kB 00:00 2024-04-15T15:12:56.330 INFO:teuthology.orchestra.run.smithi096.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-04-15T15:12:56.356 INFO:teuthology.orchestra.run.smithi096.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-04-15T15:12:56.389 INFO:teuthology.orchestra.run.smithi096.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-04-15T15:12:56.407 INFO:teuthology.orchestra.run.smithi096.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 677 kB/s | 34 kB 00:00 2024-04-15T15:12:56.431 INFO:teuthology.orchestra.run.smithi070.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 35 MB/s | 4.7 MB 00:00 2024-04-15T15:12:56.515 INFO:teuthology.orchestra.run.smithi096.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-04-15T15:12:56.523 INFO:teuthology.orchestra.run.smithi070.stdout:(56/174): postgresql-13.14-1.el9.x86_64.rpm 17 MB/s | 1.5 MB 00:00 2024-04-15T15:12:56.548 INFO:teuthology.orchestra.run.smithi096.stdout:(57/174): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-04-15T15:12:56.582 INFO:teuthology.orchestra.run.smithi070.stdout:(57/174): postgresql-contrib-13.14-1.el9.x86_64 14 MB/s | 817 kB 00:00 2024-04-15T15:12:56.582 INFO:teuthology.orchestra.run.smithi096.stdout:(58/174): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 817 kB 00:00 2024-04-15T15:12:56.599 INFO:teuthology.orchestra.run.smithi096.stdout:(59/174): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 132 kB 00:00 2024-04-15T15:12:56.624 INFO:teuthology.orchestra.run.smithi070.stdout:(58/174): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 132 kB 00:00 2024-04-15T15:12:56.733 INFO:teuthology.orchestra.run.smithi096.stdout:(60/174): postgresql-server-13.14-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-04-15T15:12:56.766 INFO:teuthology.orchestra.run.smithi096.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-04-15T15:12:56.792 INFO:teuthology.orchestra.run.smithi096.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-04-15T15:12:56.807 INFO:teuthology.orchestra.run.smithi070.stdout:(59/174): postgresql-server-13.14-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-04-15T15:12:56.883 INFO:teuthology.orchestra.run.smithi070.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-04-15T15:12:56.909 INFO:teuthology.orchestra.run.smithi096.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 41 MB/s | 5.8 MB 00:00 2024-04-15T15:12:56.934 INFO:teuthology.orchestra.run.smithi096.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-04-15T15:12:56.959 INFO:teuthology.orchestra.run.smithi096.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-04-15T15:12:56.985 INFO:teuthology.orchestra.run.smithi096.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-15T15:12:57.010 INFO:teuthology.orchestra.run.smithi096.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-04-15T15:12:57.143 INFO:teuthology.orchestra.run.smithi096.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-04-15T15:12:57.168 INFO:teuthology.orchestra.run.smithi096.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-04-15T15:12:57.194 INFO:teuthology.orchestra.run.smithi096.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-04-15T15:12:57.219 INFO:teuthology.orchestra.run.smithi096.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-04-15T15:12:57.244 INFO:teuthology.orchestra.run.smithi096.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-04-15T15:12:57.270 INFO:teuthology.orchestra.run.smithi096.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-04-15T15:12:57.295 INFO:teuthology.orchestra.run.smithi096.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-04-15T15:12:57.320 INFO:teuthology.orchestra.run.smithi096.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-04-15T15:12:57.324 INFO:teuthology.orchestra.run.smithi070.stdout:(61/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8.x 6.6 MB/s | 49 MB 00:07 2024-04-15T15:12:57.345 INFO:teuthology.orchestra.run.smithi096.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 954 kB/s | 47 kB 00:00 2024-04-15T15:12:57.349 INFO:teuthology.orchestra.run.smithi070.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 166 kB/s | 78 kB 00:00 2024-04-15T15:12:57.371 INFO:teuthology.orchestra.run.smithi096.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 858 kB/s | 43 kB 00:00 2024-04-15T15:12:57.554 INFO:teuthology.orchestra.run.smithi096.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 211 kB/s | 39 kB 00:00 2024-04-15T15:12:57.667 INFO:teuthology.orchestra.run.smithi070.stdout:(63/174): ceph-common-19.0.0-2861.g9bf1c4ba.el8 2.1 MB/s | 21 MB 00:09 2024-04-15T15:12:57.742 INFO:teuthology.orchestra.run.smithi070.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-04-15T15:12:57.763 INFO:teuthology.orchestra.run.smithi096.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-15T15:12:57.783 INFO:teuthology.orchestra.run.smithi070.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 57 kB/s | 24 kB 00:00 2024-04-15T15:12:57.788 INFO:teuthology.orchestra.run.smithi096.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 177 kB/s | 41 kB 00:00 2024-04-15T15:12:57.809 INFO:teuthology.orchestra.run.smithi070.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 128 kB/s | 18 kB 00:00 2024-04-15T15:12:57.813 INFO:teuthology.orchestra.run.smithi096.stdout:(81/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-04-15T15:12:57.834 INFO:teuthology.orchestra.run.smithi070.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-04-15T15:12:57.839 INFO:teuthology.orchestra.run.smithi096.stdout:(82/174): qatlib-service-24.02.0-1.el9.x86_64.r 700 kB/s | 35 kB 00:00 2024-04-15T15:12:57.859 INFO:teuthology.orchestra.run.smithi070.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 421 kB/s | 32 kB 00:00 2024-04-15T15:12:57.864 INFO:teuthology.orchestra.run.smithi096.stdout:(83/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 921 kB/s | 46 kB 00:00 2024-04-15T15:12:57.890 INFO:teuthology.orchestra.run.smithi096.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-04-15T15:12:57.915 INFO:teuthology.orchestra.run.smithi096.stdout:(85/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-04-15T15:12:57.941 INFO:teuthology.orchestra.run.smithi096.stdout:(86/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-15T15:12:57.951 INFO:teuthology.orchestra.run.smithi070.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-04-15T15:12:57.966 INFO:teuthology.orchestra.run.smithi096.stdout:(87/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-04-15T15:12:57.976 INFO:teuthology.orchestra.run.smithi070.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-04-15T15:12:57.990 INFO:teuthology.orchestra.run.smithi096.stdout:(88/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-04-15T15:12:58.002 INFO:teuthology.orchestra.run.smithi070.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-04-15T15:12:58.016 INFO:teuthology.orchestra.run.smithi096.stdout:(89/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-04-15T15:12:58.027 INFO:teuthology.orchestra.run.smithi070.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 913 kB/s | 69 kB 00:00 2024-04-15T15:12:58.052 INFO:teuthology.orchestra.run.smithi070.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 825 kB/s | 41 kB 00:00 2024-04-15T15:12:58.058 INFO:teuthology.orchestra.run.smithi096.stdout:(90/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-04-15T15:12:58.077 INFO:teuthology.orchestra.run.smithi070.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-04-15T15:12:58.083 INFO:teuthology.orchestra.run.smithi096.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-04-15T15:12:58.103 INFO:teuthology.orchestra.run.smithi070.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-04-15T15:12:58.108 INFO:teuthology.orchestra.run.smithi096.stdout:(92/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-04-15T15:12:58.120 INFO:teuthology.orchestra.run.smithi070.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-04-15T15:12:58.134 INFO:teuthology.orchestra.run.smithi096.stdout:(93/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-04-15T15:12:58.145 INFO:teuthology.orchestra.run.smithi070.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-04-15T15:12:58.162 INFO:teuthology.orchestra.run.smithi070.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 731 kB/s | 43 kB 00:00 2024-04-15T15:12:58.209 INFO:teuthology.orchestra.run.smithi096.stdout:(94/174): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2024-04-15T15:12:58.259 INFO:teuthology.orchestra.run.smithi096.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-04-15T15:12:58.301 INFO:teuthology.orchestra.run.smithi096.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-15T15:12:58.313 INFO:teuthology.orchestra.run.smithi070.stdout:(79/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 279 kB/s | 41 kB 00:00 2024-04-15T15:12:58.335 INFO:teuthology.orchestra.run.smithi096.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-04-15T15:12:58.337 INFO:teuthology.orchestra.run.smithi070.stdout:(80/174): python3-toml-0.10.2-6.el9.noarch.rpm 224 kB/s | 39 kB 00:00 2024-04-15T15:12:58.352 INFO:teuthology.orchestra.run.smithi096.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-15T15:12:58.385 INFO:teuthology.orchestra.run.smithi096.stdout:(99/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-04-15T15:12:58.469 INFO:teuthology.orchestra.run.smithi096.stdout:(100/174): python3-devel-3.9.17-2.el9.x86_64.rp 2.9 MB/s | 245 kB 00:00 2024-04-15T15:12:58.506 INFO:teuthology.orchestra.run.smithi070.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-04-15T15:12:58.529 INFO:teuthology.orchestra.run.smithi070.stdout:(82/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.0 MB/s | 220 kB 00:00 2024-04-15T15:12:58.546 INFO:teuthology.orchestra.run.smithi070.stdout:(83/174): qatlib-service-24.02.0-1.el9.x86_64.r 169 kB/s | 35 kB 00:00 2024-04-15T15:12:58.553 INFO:teuthology.orchestra.run.smithi096.stdout:(101/174): python3-jinja2-2.11.3-5.el9.noarch.r 2.9 MB/s | 248 kB 00:00 2024-04-15T15:12:58.571 INFO:teuthology.orchestra.run.smithi070.stdout:(84/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-04-15T15:12:58.588 INFO:teuthology.orchestra.run.smithi070.stdout:(85/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 790 kB/s | 46 kB 00:00 2024-04-15T15:12:58.605 INFO:teuthology.orchestra.run.smithi070.stdout:(86/174): uuid-1.6.2-55.el9.x86_64.rpm 959 kB/s | 56 kB 00:00 2024-04-15T15:12:58.622 INFO:teuthology.orchestra.run.smithi070.stdout:(87/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-04-15T15:12:58.628 INFO:teuthology.orchestra.run.smithi096.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-15T15:12:58.639 INFO:teuthology.orchestra.run.smithi070.stdout:(88/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-04-15T15:12:58.664 INFO:teuthology.orchestra.run.smithi070.stdout:(89/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-04-15T15:12:58.687 INFO:teuthology.orchestra.run.smithi096.stdout:(103/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-04-15T15:12:58.689 INFO:teuthology.orchestra.run.smithi070.stdout:(90/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-04-15T15:12:58.715 INFO:teuthology.orchestra.run.smithi070.stdout:(91/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-04-15T15:12:58.749 INFO:teuthology.orchestra.run.smithi070.stdout:(92/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-04-15T15:12:58.762 INFO:teuthology.orchestra.run.smithi096.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-04-15T15:12:58.773 INFO:teuthology.orchestra.run.smithi070.stdout:(93/174): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 124 kB 00:00 2024-04-15T15:12:58.799 INFO:teuthology.orchestra.run.smithi070.stdout:(94/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-04-15T15:12:58.866 INFO:teuthology.orchestra.run.smithi070.stdout:(95/174): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-04-15T15:12:58.879 INFO:teuthology.orchestra.run.smithi096.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 4.4 MB/s | 863 kB 00:00 2024-04-15T15:12:58.891 INFO:teuthology.orchestra.run.smithi070.stdout:(96/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-15T15:12:58.950 INFO:teuthology.orchestra.run.smithi070.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-15T15:12:58.975 INFO:teuthology.orchestra.run.smithi070.stdout:(98/174): elfutils-libelf-devel-0.190-2.el9.x86 258 kB/s | 21 kB 00:00 2024-04-15T15:12:58.996 INFO:teuthology.orchestra.run.smithi096.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 2.3 MB/s | 548 kB 00:00 2024-04-15T15:12:59.000 INFO:teuthology.orchestra.run.smithi070.stdout:(99/174): python3-requests-2.25.1-8.el9.noarch. 932 kB/s | 125 kB 00:00 2024-04-15T15:12:59.046 INFO:teuthology.orchestra.run.smithi096.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-15T15:12:59.067 INFO:teuthology.orchestra.run.smithi070.stdout:(100/174): libuuid-devel-2.37.4-18.el9.x86_64.r 217 kB/s | 25 kB 00:00 2024-04-15T15:12:59.163 INFO:teuthology.orchestra.run.smithi096.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-04-15T15:12:59.213 INFO:teuthology.orchestra.run.smithi096.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-04-15T15:12:59.250 INFO:teuthology.orchestra.run.smithi070.stdout:(101/174): openssl-devel-3.0.7-27.el9.x86_64.rp 15 MB/s | 4.1 MB 00:00 2024-04-15T15:12:59.264 INFO:teuthology.orchestra.run.smithi096.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2024-04-15T15:12:59.276 INFO:teuthology.orchestra.run.smithi070.stdout:(102/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.2 MB/s | 248 kB 00:00 2024-04-15T15:12:59.301 INFO:teuthology.orchestra.run.smithi070.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-04-15T15:12:59.331 INFO:teuthology.orchestra.run.smithi096.stdout:(111/174): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-15T15:12:59.343 INFO:teuthology.orchestra.run.smithi070.stdout:(104/174): python3-devel-3.9.17-2.el9.x86_64.rp 715 kB/s | 245 kB 00:00 2024-04-15T15:12:59.356 INFO:teuthology.orchestra.run.smithi096.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-04-15T15:12:59.382 INFO:teuthology.orchestra.run.smithi096.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 2.9 MB/s | 151 kB 00:00 2024-04-15T15:12:59.440 INFO:teuthology.orchestra.run.smithi096.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-04-15T15:12:59.474 INFO:teuthology.orchestra.run.smithi096.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-15T15:12:59.478 INFO:teuthology.orchestra.run.smithi070.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 110 kB/s | 22 kB 00:00 2024-04-15T15:12:59.499 INFO:teuthology.orchestra.run.smithi096.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 741 kB/s | 43 kB 00:00 2024-04-15T15:12:59.524 INFO:teuthology.orchestra.run.smithi096.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 641 kB/s | 32 kB 00:00 2024-04-15T15:12:59.549 INFO:teuthology.orchestra.run.smithi096.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-15T15:12:59.575 INFO:teuthology.orchestra.run.smithi096.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-04-15T15:12:59.600 INFO:teuthology.orchestra.run.smithi096.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-04-15T15:12:59.625 INFO:teuthology.orchestra.run.smithi096.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-15T15:12:59.651 INFO:teuthology.orchestra.run.smithi096.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-04-15T15:12:59.676 INFO:teuthology.orchestra.run.smithi096.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-04-15T15:12:59.718 INFO:teuthology.orchestra.run.smithi096.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 3.4 MB/s | 144 kB 00:00 2024-04-15T15:12:59.760 INFO:teuthology.orchestra.run.smithi096.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-04-15T15:12:59.777 INFO:teuthology.orchestra.run.smithi070.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-04-15T15:12:59.785 INFO:teuthology.orchestra.run.smithi096.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-04-15T15:12:59.810 INFO:teuthology.orchestra.run.smithi096.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-04-15T15:12:59.835 INFO:teuthology.orchestra.run.smithi096.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-15T15:12:59.869 INFO:teuthology.orchestra.run.smithi096.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-04-15T15:12:59.886 INFO:teuthology.orchestra.run.smithi096.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-04-15T15:12:59.938 INFO:teuthology.orchestra.run.smithi096.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-04-15T15:12:59.953 INFO:teuthology.orchestra.run.smithi096.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 690 kB/s | 46 kB 00:00 2024-04-15T15:12:59.987 INFO:teuthology.orchestra.run.smithi096.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-15T15:13:00.004 INFO:teuthology.orchestra.run.smithi096.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-04-15T15:13:00.038 INFO:teuthology.orchestra.run.smithi096.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-04-15T15:13:00.054 INFO:teuthology.orchestra.run.smithi096.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-15T15:13:00.088 INFO:teuthology.orchestra.run.smithi096.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-04-15T15:13:00.102 INFO:teuthology.orchestra.run.smithi070.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 722 kB/s | 548 kB 00:00 2024-04-15T15:13:00.113 INFO:teuthology.orchestra.run.smithi096.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 482 kB/s | 28 kB 00:00 2024-04-15T15:13:00.127 INFO:teuthology.orchestra.run.smithi070.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 495 kB/s | 309 kB 00:00 2024-04-15T15:13:00.138 INFO:teuthology.orchestra.run.smithi096.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 613 kB/s | 31 kB 00:00 2024-04-15T15:13:00.153 INFO:teuthology.orchestra.run.smithi070.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-04-15T15:13:00.172 INFO:teuthology.orchestra.run.smithi096.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.1 MB/s | 188 kB 00:00 2024-04-15T15:13:00.197 INFO:teuthology.orchestra.run.smithi096.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-04-15T15:13:00.228 INFO:teuthology.orchestra.run.smithi070.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-04-15T15:13:00.231 INFO:teuthology.orchestra.run.smithi096.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-04-15T15:13:00.256 INFO:teuthology.orchestra.run.smithi096.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 868 kB/s | 51 kB 00:00 2024-04-15T15:13:00.282 INFO:teuthology.orchestra.run.smithi096.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-15T15:13:00.307 INFO:teuthology.orchestra.run.smithi096.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.7 MB/s | 90 kB 00:00 2024-04-15T15:13:00.336 INFO:teuthology.orchestra.run.smithi070.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 624 kB/s | 67 kB 00:00 2024-04-15T15:13:00.357 INFO:teuthology.orchestra.run.smithi096.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.5 MB/s | 427 kB 00:00 2024-04-15T15:13:00.362 INFO:teuthology.orchestra.run.smithi070.stdout:(112/174): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-04-15T15:13:00.383 INFO:teuthology.orchestra.run.smithi096.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-15T15:13:00.408 INFO:teuthology.orchestra.run.smithi096.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-15T15:13:00.450 INFO:teuthology.orchestra.run.smithi096.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-04-15T15:13:00.462 INFO:teuthology.orchestra.run.smithi070.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-15T15:13:00.529 INFO:teuthology.orchestra.run.smithi070.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-04-15T15:13:00.629 INFO:teuthology.orchestra.run.smithi070.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-04-15T15:13:00.654 INFO:teuthology.orchestra.run.smithi070.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 2.8 MB/s | 548 kB 00:00 2024-04-15T15:13:00.730 INFO:teuthology.orchestra.run.smithi070.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-04-15T15:13:00.755 INFO:teuthology.orchestra.run.smithi070.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-04-15T15:13:00.789 INFO:teuthology.orchestra.run.smithi070.stdout:(119/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-04-15T15:13:00.830 INFO:teuthology.orchestra.run.smithi070.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-15T15:13:00.872 INFO:teuthology.orchestra.run.smithi070.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-04-15T15:13:00.931 INFO:teuthology.orchestra.run.smithi070.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 352 kB/s | 35 kB 00:00 2024-04-15T15:13:00.956 INFO:teuthology.orchestra.run.smithi070.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.8 MB/s | 303 kB 00:00 2024-04-15T15:13:00.982 INFO:teuthology.orchestra.run.smithi070.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-04-15T15:13:01.065 INFO:teuthology.orchestra.run.smithi070.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-04-15T15:13:01.082 INFO:teuthology.orchestra.run.smithi070.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-04-15T15:13:01.099 INFO:teuthology.orchestra.run.smithi070.stdout:(127/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-15T15:13:01.157 INFO:teuthology.orchestra.run.smithi070.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-15T15:13:01.174 INFO:teuthology.orchestra.run.smithi070.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2024-04-15T15:13:01.191 INFO:teuthology.orchestra.run.smithi070.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-15T15:13:01.250 INFO:teuthology.orchestra.run.smithi070.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-15T15:13:01.283 INFO:teuthology.orchestra.run.smithi070.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-04-15T15:13:01.342 INFO:teuthology.orchestra.run.smithi070.stdout:(133/174): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-04-15T15:13:01.369 INFO:teuthology.orchestra.run.smithi070.stdout:(134/174): python3-more-itertools-8.12.0-2.el9. 672 kB/s | 79 kB 00:00 2024-04-15T15:13:01.393 INFO:teuthology.orchestra.run.smithi070.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 529 kB/s | 58 kB 00:00 2024-04-15T15:13:01.452 INFO:teuthology.orchestra.run.smithi070.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-15T15:13:01.475 INFO:teuthology.orchestra.run.smithi096.stdout:(150/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 3.1 MB/s | 3.1 MB 00:01 2024-04-15T15:13:01.477 INFO:teuthology.orchestra.run.smithi070.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 150 kB/s | 16 kB 00:00 2024-04-15T15:13:01.493 INFO:teuthology.orchestra.run.smithi070.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 895 kB/s | 90 kB 00:00 2024-04-15T15:13:01.559 INFO:teuthology.orchestra.run.smithi096.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 28 MB/s | 2.3 MB 00:00 2024-04-15T15:13:01.577 INFO:teuthology.orchestra.run.smithi070.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 308 kB/s | 31 kB 00:00 2024-04-15T15:13:01.602 INFO:teuthology.orchestra.run.smithi070.stdout:(140/174): python3-pyperclip-1.8.2-2.el9.noarch 187 kB/s | 28 kB 00:00 2024-04-15T15:13:01.626 INFO:teuthology.orchestra.run.smithi096.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 27 MB/s | 1.8 MB 00:00 2024-04-15T15:13:01.627 INFO:teuthology.orchestra.run.smithi070.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-04-15T15:13:01.676 INFO:teuthology.orchestra.run.smithi096.stdout:(153/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 2.7 MB/s | 3.4 MB 00:01 2024-04-15T15:13:01.678 INFO:teuthology.orchestra.run.smithi070.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-04-15T15:13:01.701 INFO:teuthology.orchestra.run.smithi096.stdout:(154/174): NetworkManager-team-1.47.4-1.el9.x86 542 kB/s | 41 kB 00:00 2024-04-15T15:13:01.703 INFO:teuthology.orchestra.run.smithi070.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 357 kB/s | 36 kB 00:00 2024-04-15T15:13:01.726 INFO:teuthology.orchestra.run.smithi096.stdout:(155/174): NetworkManager-tui-1.47.4-1.el9.x86_ 4.8 MB/s | 248 kB 00:00 2024-04-15T15:13:01.729 INFO:teuthology.orchestra.run.smithi070.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-04-15T15:13:01.751 INFO:teuthology.orchestra.run.smithi096.stdout:(156/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-04-15T15:13:01.779 INFO:teuthology.orchestra.run.smithi070.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-15T15:13:01.802 INFO:teuthology.orchestra.run.smithi096.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 957 kB/s | 71 kB 00:00 2024-04-15T15:13:01.812 INFO:teuthology.orchestra.run.smithi070.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 824 kB/s | 90 kB 00:00 2024-04-15T15:13:01.827 INFO:teuthology.orchestra.run.smithi096.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 484 kB/s | 36 kB 00:00 2024-04-15T15:13:01.862 INFO:teuthology.orchestra.run.smithi070.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-04-15T15:13:01.886 INFO:teuthology.orchestra.run.smithi096.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.2 MB/s | 192 kB 00:00 2024-04-15T15:13:01.888 INFO:teuthology.orchestra.run.smithi070.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-04-15T15:13:01.919 INFO:teuthology.orchestra.run.smithi096.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-04-15T15:13:01.921 INFO:teuthology.orchestra.run.smithi070.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-04-15T15:13:01.945 INFO:teuthology.orchestra.run.smithi096.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-04-15T15:13:02.003 INFO:teuthology.orchestra.run.smithi096.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 108 kB 00:00 2024-04-15T15:13:02.028 INFO:teuthology.orchestra.run.smithi096.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-04-15T15:13:02.080 INFO:teuthology.orchestra.run.smithi070.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-15T15:13:02.087 INFO:teuthology.orchestra.run.smithi096.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-04-15T15:13:02.112 INFO:teuthology.orchestra.run.smithi096.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 764 kB/s | 64 kB 00:00 2024-04-15T15:13:02.154 INFO:teuthology.orchestra.run.smithi096.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 430 kB/s | 29 kB 00:00 2024-04-15T15:13:02.183 INFO:teuthology.orchestra.run.smithi070.stdout:(151/174): NetworkManager-1.47.4-1.el9.x86_64.r 22 MB/s | 2.3 MB 00:00 2024-04-15T15:13:02.255 INFO:teuthology.orchestra.run.smithi096.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-04-15T15:13:02.255 INFO:teuthology.orchestra.run.smithi070.stdout:(152/174): NetworkManager-libnm-1.47.4-1.el9.x8 25 MB/s | 1.8 MB 00:00 2024-04-15T15:13:02.281 INFO:teuthology.orchestra.run.smithi070.stdout:(153/174): NetworkManager-team-1.47.4-1.el9.x86 1.6 MB/s | 41 kB 00:00 2024-04-15T15:13:02.323 INFO:teuthology.orchestra.run.smithi070.stdout:(154/174): NetworkManager-tui-1.47.4-1.el9.x86_ 5.8 MB/s | 248 kB 00:00 2024-04-15T15:13:02.364 INFO:teuthology.orchestra.run.smithi070.stdout:(155/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-04-15T15:13:02.371 INFO:teuthology.orchestra.run.smithi096.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 10 MB/s | 2.2 MB 00:00 2024-04-15T15:13:02.440 INFO:teuthology.orchestra.run.smithi070.stdout:(156/174): librados2-19.0.0-2861.g9bf1c4ba.el8. 6.2 MB/s | 3.4 MB 00:00 2024-04-15T15:13:02.465 INFO:teuthology.orchestra.run.smithi070.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 712 kB/s | 71 kB 00:00 2024-04-15T15:13:02.480 INFO:teuthology.orchestra.run.smithi096.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 10 MB/s | 2.3 MB 00:00 2024-04-15T15:13:02.505 INFO:teuthology.orchestra.run.smithi096.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-04-15T15:13:02.515 INFO:teuthology.orchestra.run.smithi070.stdout:(158/174): librbd1-19.0.0-2861.g9bf1c4ba.el8.x8 5.3 MB/s | 3.1 MB 00:00 2024-04-15T15:13:02.549 INFO:teuthology.orchestra.run.smithi070.stdout:(159/174): elfutils-debuginfod-client-0.190-2.e 336 kB/s | 36 kB 00:00 2024-04-15T15:13:02.556 INFO:teuthology.orchestra.run.smithi096.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 628 kB/s | 31 kB 00:00 2024-04-15T15:13:02.607 INFO:teuthology.orchestra.run.smithi070.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.3 MB/s | 192 kB 00:00 2024-04-15T15:13:02.631 INFO:teuthology.orchestra.run.smithi096.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 991 kB/s | 74 kB 00:00 2024-04-15T15:13:02.632 INFO:teuthology.orchestra.run.smithi070.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 276 kB/s | 23 kB 00:00 2024-04-15T15:13:02.666 INFO:teuthology.orchestra.run.smithi070.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-04-15T15:13:02.673 INFO:teuthology.orchestra.run.smithi096.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-04-15T15:13:02.700 INFO:teuthology.orchestra.run.smithi070.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 108 kB 00:00 2024-04-15T15:13:02.734 INFO:teuthology.orchestra.run.smithi070.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-04-15T15:13:02.775 INFO:teuthology.orchestra.run.smithi070.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-04-15T15:13:02.801 INFO:teuthology.orchestra.run.smithi070.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 636 kB/s | 64 kB 00:00 2024-04-15T15:13:02.826 INFO:teuthology.orchestra.run.smithi070.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 312 kB/s | 29 kB 00:00 2024-04-15T15:13:02.935 INFO:teuthology.orchestra.run.smithi070.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 7.6 MB/s | 1.2 MB 00:00 2024-04-15T15:13:03.018 INFO:teuthology.orchestra.run.smithi070.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 11 MB/s | 2.2 MB 00:00 2024-04-15T15:13:03.077 INFO:teuthology.orchestra.run.smithi070.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 9.1 MB/s | 2.3 MB 00:00 2024-04-15T15:13:03.102 INFO:teuthology.orchestra.run.smithi070.stdout:(171/174): util-linux-core-2.37.4-18.el9.x86_64 2.7 MB/s | 465 kB 00:00 2024-04-15T15:13:03.127 INFO:teuthology.orchestra.run.smithi070.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-04-15T15:13:03.211 INFO:teuthology.orchestra.run.smithi070.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 686 kB/s | 74 kB 00:00 2024-04-15T15:13:03.244 INFO:teuthology.orchestra.run.smithi070.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.4 MB/s | 1.1 MB 00:00 2024-04-15T15:13:03.251 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:13:03.251 INFO:teuthology.orchestra.run.smithi070.stdout:Total 14 MB/s | 228 MB 00:15 2024-04-15T15:13:03.617 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-04-15T15:13:03.889 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-04-15T15:13:03.889 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-04-15T15:13:05.911 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-04-15T15:13:05.911 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-04-15T15:13:07.223 INFO:teuthology.orchestra.run.smithi096.stdout:(174/174): ceph-test-19.0.0-2861.g9bf1c4ba.el8. 3.2 MB/s | 49 MB 00:15 2024-04-15T15:13:07.227 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-04-15T15:13:07.227 INFO:teuthology.orchestra.run.smithi096.stdout:Total 12 MB/s | 228 MB 00:18 2024-04-15T15:13:07.612 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-04-15T15:13:07.884 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-04-15T15:13:07.884 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-04-15T15:13:10.037 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-04-15T15:13:10.038 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-04-15T15:13:12.859 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-04-15T15:13:12.947 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:13:12.974 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:13:13.054 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:13.064 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:13:13.349 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:13:13.382 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:13:13.407 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:13:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:13:13.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:13:13.850 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:13.956 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:13.984 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:14.066 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:14.098 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:14.135 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:13:14.338 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:13:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:13:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:14.526 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:14.548 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:14.614 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:14.661 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:13:14.684 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:14.699 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:13:15.148 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:13:15.273 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:15.433 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:15.492 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:15.705 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:15.817 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:13:15.909 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:13:15.942 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:13:15.966 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:13:16.010 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:13:16.084 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:13:16.138 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:13:16.176 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:13:16.240 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:13:16.275 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:13:16.319 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:13:16.352 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:13:16.379 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:13:16.449 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:13:16.484 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:13:16.527 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:13:16.556 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:13:16.625 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:13:16.679 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:13:16.753 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:13:16.883 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:13:16.940 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:13:16.964 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-04-15T15:13:17.043 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-04-15T15:13:17.069 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-04-15T15:13:17.140 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:17.207 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-04-15T15:13:17.398 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-04-15T15:13:17.454 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:13:17.454 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-04-15T15:13:17.495 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-04-15T15:13:17.519 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-04-15T15:13:17.535 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:13:17.576 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-04-15T15:13:17.615 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:13:17.655 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:17.664 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:17.698 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:13:17.735 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:13:17.773 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:13:17.804 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:13:17.901 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-04-15T15:13:17.965 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:18.087 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:13:18.101 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:13:18.116 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:18.207 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:13:18.237 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:13:18.275 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-04-15T15:13:18.297 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:13:18.430 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:13:18.460 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-04-15T15:13:18.518 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-04-15T15:13:18.546 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:18.557 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:13:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:13:18.672 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:18.692 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:13:18.735 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:13:18.737 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 17/198 2024-04-15T15:13:18.776 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-04-15T15:13:18.790 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:13:18.799 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:18.815 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-04-15T15:13:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:13:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:13:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:13:18.980 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:13:19.009 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:19.015 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:13:19.039 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-04-15T15:13:19.058 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:13:19.078 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-04-15T15:13:19.101 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:13:19.234 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:13:19.262 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-04-15T15:13:19.298 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:13:19.329 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:13:19.370 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:13:19.391 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:19.443 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 23/198 2024-04-15T15:13:19.464 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:13:19.528 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:13:19.569 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:13:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:13:19.619 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:13:19.695 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:19.697 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:13:19.819 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:13:19.912 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:19.935 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:13:19.977 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:13:20.025 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:13:20.081 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:13:20.151 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 24/198 2024-04-15T15:13:20.264 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:13:20.310 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-04-15T15:13:20.314 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:13:20.406 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-04-15T15:13:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-04-15T15:13:20.442 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:13:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-04-15T15:13:20.479 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:13:20.511 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:13:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-04-15T15:13:20.542 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:20.571 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:13:20.594 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-04-15T15:13:20.605 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:13:20.644 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:13:20.648 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-04-15T15:13:20.675 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:13:20.687 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-04-15T15:13:20.706 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:13:20.751 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-04-15T15:13:20.782 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:13:20.792 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-04-15T15:13:20.816 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:13:20.838 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-04-15T15:13:20.866 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:13:20.871 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-04-15T15:13:20.899 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-04-15T15:13:20.908 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:13:20.942 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:13:20.969 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-04-15T15:13:21.011 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 39/198 2024-04-15T15:13:21.031 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:13:21.047 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 40/198 2024-04-15T15:13:21.075 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 41/198 2024-04-15T15:13:21.113 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:13:21.144 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-04-15T15:13:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:13:21.189 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-04-15T15:13:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:13:21.243 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:13:21.266 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-04-15T15:13:21.342 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:13:21.390 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:13:21.398 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-04-15T15:13:21.451 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-04-15T15:13:21.974 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-04-15T15:13:22.054 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-04-15T15:13:22.117 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:13:22.164 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-04-15T15:13:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:13:22.216 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:22.227 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-04-15T15:13:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:13:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-04-15T15:13:22.278 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:13:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-04-15T15:13:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 53/198 2024-04-15T15:13:22.366 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:13:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-04-15T15:13:22.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:13:22.573 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:13:22.603 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:13:22.633 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-04-15T15:13:22.684 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:13:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-04-15T15:13:22.943 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-04-15T15:13:22.987 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-04-15T15:13:23.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-04-15T15:13:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-04-15T15:13:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-04-15T15:13:23.301 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-04-15T15:13:23.323 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:13:23.332 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-04-15T15:13:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-04-15T15:13:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-04-15T15:13:23.436 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:13:23.460 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-04-15T15:13:23.503 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-04-15T15:13:23.545 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-04-15T15:13:23.680 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-04-15T15:13:23.726 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-04-15T15:13:23.756 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-04-15T15:13:23.798 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-04-15T15:13:23.892 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-04-15T15:13:23.923 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-04-15T15:13:23.955 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-04-15T15:13:23.981 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-04-15T15:13:24.006 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-04-15T15:13:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-04-15T15:13:24.224 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-04-15T15:13:24.323 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-04-15T15:13:24.398 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-04-15T15:13:24.446 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-04-15T15:13:24.502 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-04-15T15:13:24.657 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:13:24.685 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-04-15T15:13:24.727 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-04-15T15:13:24.732 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:13:24.763 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:13:24.794 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:13:24.822 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:13:24.865 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-04-15T15:13:24.917 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-04-15T15:13:24.949 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-04-15T15:13:24.980 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:13:25.008 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-04-15T15:13:25.042 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-04-15T15:13:25.081 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-04-15T15:13:25.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-04-15T15:13:25.134 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:13:25.143 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-04-15T15:13:25.160 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:25.223 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-04-15T15:13:25.250 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:25.254 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-04-15T15:13:25.282 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:13:25.287 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-04-15T15:13:25.318 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:13:25.328 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-04-15T15:13:25.383 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-04-15T15:13:25.469 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-04-15T15:13:25.563 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-04-15T15:13:25.603 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-04-15T15:13:25.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-04-15T15:13:25.673 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-04-15T15:13:25.772 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-04-15T15:13:25.785 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:13:25.828 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-04-15T15:13:25.913 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:25.984 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:26.007 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:13:26.554 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-04-15T15:13:26.600 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-04-15T15:13:26.634 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-04-15T15:13:26.682 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-04-15T15:13:26.770 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-04-15T15:13:26.806 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-04-15T15:13:26.982 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-04-15T15:13:27.012 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-04-15T15:13:27.090 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-04-15T15:13:27.741 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-04-15T15:13:27.860 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-04-15T15:13:28.239 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:28.833 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:28.873 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:28.967 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:28.968 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:13:28.968 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:13:28.968 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:29.008 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:29.105 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-04-15T15:13:29.228 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-04-15T15:13:29.262 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-04-15T15:13:29.285 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-04-15T15:13:29.312 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-04-15T15:13:29.623 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-04-15T15:13:29.654 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:29.724 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 124/198 2024-04-15T15:13:29.756 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 125/198 2024-04-15T15:13:29.792 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-04-15T15:13:30.254 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-04-15T15:13:30.279 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:30.350 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 128/198 2024-04-15T15:13:30.381 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 129/198 2024-04-15T15:13:32.585 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:32.627 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:33.162 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 130/198 2024-04-15T15:13:33.198 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:33.296 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 131/198 2024-04-15T15:13:33.297 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-15T15:13:33.297 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-15T15:13:33.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:33.340 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-04-15T15:13:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-04-15T15:13:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-04-15T15:13:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-04-15T15:13:40.994 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:41.155 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:41.171 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:41.171 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:41.171 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:13:41.171 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:42.659 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:42.685 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:42.917 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:42.940 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:42.940 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:42.940 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:13:42.940 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:42.941 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:42.941 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:43.376 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:43.399 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:43.399 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:43.399 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:13:43.399 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:43.400 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:43.400 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:43.487 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:13:43.931 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:43.960 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:43.992 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:13:44.053 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:13:44.083 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:45.118 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:45.160 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 132/198 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-04-15T15:13:45.406 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:45.573 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:45.588 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 133/198 2024-04-15T15:13:45.588 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:45.589 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-15T15:13:45.589 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:46.129 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:46.152 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:46.252 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:46.424 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:13:46.454 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:46.477 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:46.477 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:46.478 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:13:46.478 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:46.478 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:46.478 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:46.521 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:46.533 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:46.581 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:13:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 134/198 2024-04-15T15:13:47.086 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.086 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-15T15:13:47.086 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:47.087 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-15T15:13:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:47.318 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:47.341 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 135/198 2024-04-15T15:13:47.341 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.341 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-15T15:13:47.341 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:47.342 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-15T15:13:47.342 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:47.462 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:13:47.498 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:47.523 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:47.566 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:47.588 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:47.588 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.589 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:13:47.589 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:47.650 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:47.672 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:47.673 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.673 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:13:47.673 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:47.673 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:47.673 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:47.777 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 136/198 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-15T15:13:47.800 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:47.895 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-04-15T15:13:47.975 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:47.998 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:47.999 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:47.999 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:13:47.999 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:47.999 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:47.999 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:13:48.315 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:48.344 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 138/198 2024-04-15T15:13:48.376 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 139/198 2024-04-15T15:13:48.437 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 140/198 2024-04-15T15:13:48.467 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:49.496 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 141/198 2024-04-15T15:13:49.536 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:50.500 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 142/198 2024-04-15T15:13:50.528 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:50.630 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 143/198 2024-04-15T15:13:50.791 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 144/198 2024-04-15T15:13:50.821 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:50.845 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 145/198 2024-04-15T15:13:50.846 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:50.846 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-15T15:13:50.846 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:50.846 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-15T15:13:50.846 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:50.897 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:50.909 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 146/198 2024-04-15T15:13:50.956 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 147/198 2024-04-15T15:13:51.827 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-04-15T15:13:51.858 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:51.881 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 149/198 2024-04-15T15:13:51.881 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:51.881 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-15T15:13:51.882 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:51.882 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-15T15:13:51.882 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:51.925 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:51.948 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 150/198 2024-04-15T15:13:51.948 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:51.948 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-15T15:13:51.948 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:52.009 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:52.032 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:13:52.032 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:52.032 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-15T15:13:52.032 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:52.033 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-15T15:13:52.033 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 152/198 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-15T15:13:52.358 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:13:52.564 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:13:52.635 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:13:52.662 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:13:52.853 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:13:52.970 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:13:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:13:53.071 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:13:53.127 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:13:53.187 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:13:53.220 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:13:53.260 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:13:53.290 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:13:53.346 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:13:53.406 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:13:53.438 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:13:54.091 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:54.128 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:54.149 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:54.970 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:13:55.057 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:13:55.098 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:13:55.180 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:13:55.211 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:13:55.236 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:13:55.261 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:13:55.288 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:13:55.288 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:55.384 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:55.413 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:13:55.414 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:55.498 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:55.534 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:13:55.566 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:13:55.592 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:13:55.616 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:13:55.652 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:13:55.689 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:13:55.721 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:13:55.755 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:13:55.781 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:13:55.806 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:13:55.822 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:13:55.829 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:55.829 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:56.019 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:13:56.019 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:56.102 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:13:56.134 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:13:56.160 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:13:56.186 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:13:56.211 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:13:56.277 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:13:56.939 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:13:56.980 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 154/198 2024-04-15T15:13:57.011 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 155/198 2024-04-15T15:13:57.038 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-04-15T15:13:57.228 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 157/198 2024-04-15T15:13:57.348 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-04-15T15:13:57.397 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 159/198 2024-04-15T15:13:57.446 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-04-15T15:13:57.505 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-04-15T15:13:57.548 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-04-15T15:13:57.581 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 163/198 2024-04-15T15:13:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-04-15T15:13:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-team-1:1.47.4-1.el9.x86_64 165/198 2024-04-15T15:13:57.706 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : NetworkManager-tui-1:1.47.4-1.el9.x86_64 166/198 2024-04-15T15:13:57.757 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-04-15T15:13:57.783 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-04-15T15:13:58.345 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:58.391 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:58.411 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-04-15T15:13:59.252 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 170/198 2024-04-15T15:13:59.344 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-04-15T15:13:59.386 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-04-15T15:13:59.473 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-04-15T15:13:59.507 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-04-15T15:13:59.531 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-04-15T15:13:59.557 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-04-15T15:13:59.583 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-04-15T15:13:59.584 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:59.671 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-04-15T15:13:59.701 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-04-15T15:13:59.701 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:59.769 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-04-15T15:13:59.806 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-04-15T15:13:59.845 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-04-15T15:13:59.871 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-04-15T15:13:59.887 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-04-15T15:13:59.922 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-04-15T15:13:59.968 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-04-15T15:14:00.009 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-04-15T15:14:00.034 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-04-15T15:14:00.085 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-04-15T15:14:00.110 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:00.127 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-04-15T15:14:00.134 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:00.134 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-04-15T15:14:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:00.388 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-04-15T15:14:00.420 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:00.446 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-04-15T15:14:00.480 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-04-15T15:14:00.506 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-04-15T15:14:00.571 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:14:01.704 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:14:01.705 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:14:01.706 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:14:01.721 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:14:01.722 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:14:01.723 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:14:01.724 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:14:01.725 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:14:01.726 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:14:01.727 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:14:01.728 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:14:01.729 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:14:01.730 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:14:01.731 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:14:01.732 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:01.733 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:14:01.734 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:14:04.551 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:14:04.551 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:04.551 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.552 INFO:teuthology.orchestra.run.smithi070.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.553 INFO:teuthology.orchestra.run.smithi070.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.554 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.555 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:14:04.556 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.557 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:14:04.558 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:14:04.559 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:14:04.560 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:14:04.561 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:14:04.562 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:14:04.563 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:14:04.564 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:14:04.565 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:14:04.566 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:14:04.567 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:14:04.567 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:04.567 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-04-15T15:14:05.034 DEBUG:teuthology.parallel:result is None 2024-04-15T15:14:05.976 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-04-15T15:14:05.976 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 1/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 3/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 4/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c 5/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 6/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 7/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 8/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 9/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 10/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 11/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 12/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 13/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 14/198 2024-04-15T15:14:05.977 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_ 15/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 16/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 17/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 18/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86 19/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 20/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el 21/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8. 22/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_6 23/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 24/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 25/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 26/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 27/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 28/198 2024-04-15T15:14:05.978 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 29/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba. 30/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noa 31/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.n 32/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf 33/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el 34/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 35/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.e 36/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 37/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 38/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 39/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-04-15T15:14:05.979 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-04-15T15:14:05.980 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-13.14-1.el9.x86_64 58/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 59/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 60/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 61/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-04-15T15:14:05.981 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 82/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 83/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 84/198 2024-04-15T15:14:05.982 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 85/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 86/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 87/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 88/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 89/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 90/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 91/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 92/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 93/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : jq-1.6-16.el9.x86_64 95/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 96/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 97/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 98/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 99/198 2024-04-15T15:14:05.983 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 100/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 101/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 102/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 103/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-04-15T15:14:05.984 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-04-15T15:14:05.985 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-04-15T15:14:05.986 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 151/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 153/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 155/198 2024-04-15T15:14:05.987 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 156/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 157/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 158/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 159/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 160/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 161/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 162/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 163/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 164/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-04-15T15:14:05.988 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-04-15T15:14:05.989 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-04-15T15:14:05.990 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-04-15T15:14:09.037 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-team-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: NetworkManager-tui-1:1.47.4-1.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: acl-2.3.1-4.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-04-15T15:14:09.038 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-2.3.1-4.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libblkid-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libmount-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.039 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: automake-1.16.2-8.el9.noarch 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: bison-3.7.4-5.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.040 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.041 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-shell-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: cephfs-top-2:19.0.0-2861.g9bf1c4ba.el8.noarch 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: flex-2.6.4-9.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-15T15:14:09.042 INFO:teuthology.orchestra.run.smithi096.stdout: jq-1.6-16.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.043 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-15T15:14:09.044 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-13.14-1.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.045 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-15T15:14:09.046 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:09.047 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-15T15:14:09.048 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-15T15:14:09.049 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.0.0-2861.g9bf1c4ba.el8.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-3.el9.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-15T15:14:09.050 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:09.051 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-04-15T15:14:09.511 DEBUG:teuthology.parallel:result is None 2024-04-15T15:14:09.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:14:09.744 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:14:09.771 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:09.771 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:09.771 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:14:09.773 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9bf1c4ba967dd24077be6e5b87edda4ca87a6e14 2024-04-15T15:14:09.914 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-15T15:14:09.937 INFO:teuthology.orchestra.run.smithi096.stdout:19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:09.937 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2861.g9bf1c4ba.el8 2024-04-15T15:14:09.937 INFO:teuthology.task.install:The correct ceph version 19.0.0-2861.g9bf1c4ba is installed. 2024-04-15T15:14:09.939 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-15T15:14:09.939 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:09.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:14:09.976 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:09.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-15T15:14:10.008 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-15T15:14:10.009 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:10.009 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:14:10.049 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:14:10.119 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:10.119 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-04-15T15:14:10.149 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-15T15:14:10.217 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-15T15:14:10.218 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:10.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:14:10.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:14:10.321 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:10.321 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-15T15:14:10.349 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-15T15:14:10.418 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-15T15:14:10.418 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:10.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:14:10.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:14:10.518 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:10.518 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-04-15T15:14:10.546 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-15T15:14:10.614 INFO:teuthology.run_tasks:Running task ceph... 2024-04-15T15:14:10.725 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-15T15:14:10.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-04-15T15:14:10.728 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-04-15T15:14:10.756 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-15T15:14:10.756 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:14:10.798 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-15T15:14:10.825 INFO:tasks.ceph:Creating extra log directories... 2024-04-15T15:14:10.825 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:14:10.867 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-15T15:14:10.901 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-15T15:14:10.901 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-15T15:14:10.901 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-15T15:14:10.902 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:10.902 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:14:10.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:11.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:11.101 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:11.101 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-15T15:14:11.116 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:11.238 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:11.272 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-15T15:14:11.272 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-15T15:14:11.272 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121', 'branch': 'wip-lusov-quiesce-xlock', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce}', 'email': None, 'first_in_suite': False, 'job_id': '7657121', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-lusov-quiesce-xlock'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-quiesce-xlock', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 2288, 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-lusov-quiesce-xlock', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoB5nFh91vNFMaqMdEZvAO3Q5iV1zSQy++mNWxmIrkasO1wNgVOeZSJqySUWA+WXmqhQI2uCfETN2LpmTmCYXQ=', 'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJEsMQ1pqCEDgo13JQ7ZcpjjN11GlQS6KwaEcdrg7vuceBTRN9SO5GlKQAlCVhLwlf/pgcNfejnvWruRhTvlfgA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request'], 'sha1': '9bf1c4ba967dd24077be6e5b87edda4ca87a6e14', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_quiesce']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '6c637841c215537a4502385240412f1966e0faab', 'timestamp': '2024-04-15_14:46:31', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-15T15:14:11.273 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:14:11.275 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-15T15:14:11.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:14:11.330 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-04-15T15:14:11.390 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:11.390 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:14:11.412 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:14:11.412 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-15 15:13:56.608422665 +0000 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-15 15:11:59.606703171 +0000 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-15 15:11:59.606703171 +0000 2024-04-15T15:14:11.468 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-15 15:11:59.606703171 +0000 2024-04-15T15:14:11.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:14:11.535 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-15T15:14:11.535 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-15T15:14:11.535 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.00010536 s, 4.9 MB/s 2024-04-15T15:14:11.536 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:14:11.593 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-15 15:13:56.608422665 +0000 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-15 15:11:59.946692855 +0000 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-15 15:11:59.946692855 +0000 2024-04-15T15:14:11.649 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-15 15:11:59.946692855 +0000 2024-04-15T15:14:11.649 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:14:11.715 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-15T15:14:11.715 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-15T15:14:11.716 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 8.3377e-05 s, 6.1 MB/s 2024-04-15T15:14:11.716 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:14:11.772 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-15 15:13:56.608422665 +0000 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-15 15:12:00.292682355 +0000 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-15 15:12:00.292682355 +0000 2024-04-15T15:14:11.828 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-15 15:12:00.292682355 +0000 2024-04-15T15:14:11.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:14:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-15T15:14:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-15T15:14:11.895 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000100764 s, 5.1 MB/s 2024-04-15T15:14:11.896 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:14:11.952 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-04-15T15:14:12.007 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-04-15 15:13:56.609422635 +0000 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-04-15 15:12:00.633672007 +0000 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-04-15 15:12:00.633672007 +0000 2024-04-15T15:14:12.008 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-04-15 15:12:00.633672007 +0000 2024-04-15T15:14:12.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:14:12.074 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-04-15T15:14:12.074 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-04-15T15:14:12.074 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000103747 s, 4.9 MB/s 2024-04-15T15:14:12.075 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:14:12.132 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:12.133 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:12.133 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T15:14:12.152 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T15:14:12.152 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-04-15T15:14:12.210 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T15:14:12.210 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:12.210 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-15 15:14:00.930363624 +0000 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-15 15:12:00.260586949 +0000 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-15 15:12:00.260586949 +0000 2024-04-15T15:14:12.211 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-15 15:12:00.260586949 +0000 2024-04-15T15:14:12.212 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T15:14:12.280 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-15T15:14:12.280 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-15T15:14:12.281 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000275003 s, 1.9 MB/s 2024-04-15T15:14:12.282 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T15:14:12.340 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-15 15:14:00.930363624 +0000 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-15 15:12:00.619578338 +0000 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-15 15:12:00.619578338 +0000 2024-04-15T15:14:12.398 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-15 15:12:00.619578338 +0000 2024-04-15T15:14:12.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T15:14:12.466 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-15T15:14:12.466 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-15T15:14:12.466 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000202396 s, 2.5 MB/s 2024-04-15T15:14:12.467 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T15:14:12.525 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:12.582 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-15 15:14:00.930363624 +0000 2024-04-15T15:14:12.583 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-15 15:12:00.974569824 +0000 2024-04-15T15:14:12.583 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-15 15:12:00.974569824 +0000 2024-04-15T15:14:12.584 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-15 15:12:00.974569824 +0000 2024-04-15T15:14:12.584 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T15:14:12.649 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-15T15:14:12.650 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-15T15:14:12.650 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000197985 s, 2.6 MB/s 2024-04-15T15:14:12.654 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T15:14:12.712 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-04-15T15:14:12.767 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-04-15 15:14:00.931363601 +0000 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-04-15 15:12:01.312561717 +0000 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-04-15 15:12:01.312561717 +0000 2024-04-15T15:14:12.768 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-04-15 15:12:01.312561717 +0000 2024-04-15T15:14:12.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T15:14:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-04-15T15:14:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-04-15T15:14:12.835 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000171693 s, 3.0 MB/s 2024-04-15T15:14:12.836 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T15:14:12.892 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:12.893 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:14:12.893 INFO:tasks.ceph:Generating config... 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] debug client = 20 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] debug ms = 1 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] pid file = 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-15T15:14:12.894 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-15T15:14:12.894 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-15T15:14:12.894 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-15T15:14:12.895 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-15T15:14:12.896 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-15T15:14:12.896 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-15T15:14:12.897 INFO:tasks.ceph:Setting up mon.a... 2024-04-15T15:14:12.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:12.952 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.keyring 2024-04-15T15:14:12.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-15T15:14:13.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:13.030 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.70'), ('mon.b', '172.21.15.96'), ('mon.c', '[v2:172.21.15.96:3301,v1:172.21.15.96:6790]')] 2024-04-15T15:14:13.030 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.70,172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-15T15:14:13.031 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:13.031 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:14:13.086 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.70 --add b 172.21.15.96 --addv c '[v2:172.21.15.96:3301,v1:172.21.15.96:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: generated fsid 074678c8-a3ad-4fb8-89f1-2c7d457fbbe5 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:setting min_mon_release = quincy 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:epoch 0 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:fsid 074678c8-a3ad-4fb8-89f1-2c7d457fbbe5 2024-04-15T15:14:13.173 INFO:teuthology.orchestra.run.smithi070.stdout:last_changed 2024-04-15T15:14:13.172524+0000 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:created 2024-04-15T15:14:13.172524+0000 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:min_mon_release 17 (quincy) 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:election_strategy: 1 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:0: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.a 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:1: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:2: [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] mon.c 2024-04-15T15:14:13.174 INFO:teuthology.orchestra.run.smithi070.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-15T15:14:13.175 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-15T15:14:13.190 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 074678c8-a3ad-4fb8-89f1-2c7d457fbbe5... 2024-04-15T15:14:13.191 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:14:13.233 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-15T15:14:13.283 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-04-15T15:14:13.283 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-04-15T15:14:13.283 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.284 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:14:13.285 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-04-15T15:14:13.286 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = "172.21.15.70,172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790]" 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 074678c8-a3ad-4fb8-89f1-2c7d457fbbe5 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-04-15T15:14:13.287 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-04-15T15:14:13.288 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:13.289 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: osd op complaint time = 180 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:13.290 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: debug client = 20 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:13.291 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-04-15T15:14:13.292 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi070.stdout: mon down mkfs grace = 300 2024-04-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi070.stdout: mon op complaint time = 120 2024-04-15T15:14:13.293 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: client mount timeout = 600 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: debug client = 20 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = "" 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: rados mon op timeout = 900 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout: rados osd op timeout = 900 2024-04-15T15:14:13.294 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.b] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.c] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout:[mds] 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout: debug mds = 20 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout: debug mds balancer = 20 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout: mds debug frag = True 2024-04-15T15:14:13.295 INFO:teuthology.orchestra.run.smithi070.stdout: mds debug scatterstat = True 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi070.stdout: mds op complaint time = 180 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi070.stdout: mds verify scatter = True 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi070.stdout: osd op complaint time = 180 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi070.stdout: rados mon op timeout = 900 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi070.stdout: rados osd op timeout = 900 2024-04-15T15:14:13.296 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-04-15T15:14:13.297 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-04-15T15:14:13.298 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-04-15T15:14:13.299 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "172.21.15.70,172.21.15.96,[v2:172.21.15.96:3301,v1:172.21.15.96:6790]" 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = 074678c8-a3ad-4fb8-89f1-2c7d457fbbe5 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-04-15T15:14:13.300 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-04-15T15:14:13.301 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 20 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 20 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-04-15T15:14:13.302 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 10 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: osd op complaint time = 180 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-04-15T15:14:13.303 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug client = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-04-15T15:14:13.304 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-04-15T15:14:13.305 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: mon down mkfs grace = 300 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: mon op complaint time = 120 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: client mount timeout = 600 2024-04-15T15:14:13.306 INFO:teuthology.orchestra.run.smithi096.stdout: debug client = 20 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = "" 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: rados mon op timeout = 900 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: rados osd op timeout = 900 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.a] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.c] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.b] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.d] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout:[mds] 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: debug mds = 20 2024-04-15T15:14:13.307 INFO:teuthology.orchestra.run.smithi096.stdout: debug mds balancer = 20 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: mds debug frag = True 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: mds debug scatterstat = True 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: mds op complaint time = 180 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: mds verify scatter = True 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: osd op complaint time = 180 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: rados mon op timeout = 900 2024-04-15T15:14:13.308 INFO:teuthology.orchestra.run.smithi096.stdout: rados osd op timeout = 900 2024-04-15T15:14:13.309 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-15T15:14:13.310 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-15T15:14:13.407 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-15T15:14:13.407 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:13.407 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-15T15:14:13.423 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:13.423 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-15T15:14:13.477 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:14:13.478 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:13.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:14:13.478 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:13.559 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:13.559 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:13.614 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:14:13.614 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:13.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-15T15:14:13.614 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-15T15:14:13.656 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:13.656 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:13.712 INFO:tasks.ceph:Setting up mon nodes... 2024-04-15T15:14:13.712 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-15T15:14:13.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:14:13.770 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-15T15:14:13.772 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:14:13.836 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-15T15:14:13.839 INFO:tasks.ceph:Setting up mds nodes... 2024-04-15T15:14:13.839 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:14:13.896 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-15T15:14:13.899 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-15T15:14:13.927 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:14:14.023 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-15T15:14:14.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-15T15:14:14.054 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:14:14.112 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-15T15:14:14.115 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-15T15:14:14.143 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:14:14.240 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-15T15:14:14.243 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-15T15:14:14.270 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-15T15:14:14.271 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T15:14:14.316 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-15T15:14:14.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-15T15:14:14.417 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-15T15:14:14.433 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T15:14:14.477 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-15T15:14:14.495 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-15T15:14:14.538 INFO:teuthology.orchestra.run.smithi096.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-15T15:14:14.555 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-15T15:14:14.555 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-15T15:14:14.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:14.583 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:14.584 INFO:tasks.ceph:role: osd.0 2024-04-15T15:14:14.584 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:14:14.584 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:14:14.651 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:14.652 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:14.653 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:14.653 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:14.653 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:14.684 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-15T15:14:14.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-15T15:14:14.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:14.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:14.832 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:14.905 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:14.905 INFO:tasks.ceph:role: osd.1 2024-04-15T15:14:14.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:14:14.905 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:14.972 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:14.973 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:14.973 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-15T15:14:15.007 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-15T15:14:15.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:15.079 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:15.152 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:15.222 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:15.222 INFO:tasks.ceph:role: osd.2 2024-04-15T15:14:15.222 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:14:15.222 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:14:15.289 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:15.289 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:15.289 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:15.289 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:15.289 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:15.290 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:15.290 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:15.290 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:15.290 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:15.290 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:15.322 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-15T15:14:15.324 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-15T15:14:15.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:15.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:15.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:15.537 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:15.537 INFO:tasks.ceph:role: osd.3 2024-04-15T15:14:15.537 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:14:15.537 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:14:15.605 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:15.605 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:15.606 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:15.638 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-04-15T15:14:15.640 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-04-15T15:14:15.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:15.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:15.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:15.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-15T15:14:15.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-15T15:14:15.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:15.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:15.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-15T15:14:15.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:15.879+0000 7f8682d24640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-15T15:14:18.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-15T15:14:18.758 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:18.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-15T15:14:18.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:18.853+0000 7f6d1f210640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-15T15:14:21.700 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-15T15:14:21.732 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.824+0000 7f621e210640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.825+0000 7f621e210640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.825+0000 7f621e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.825+0000 7f621e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.825+0000 7f621e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-15T15:14:21.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:21.825+0000 7f621e210640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-15T15:14:24.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-15T15:14:24.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:24.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-15T15:14:24.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-15T15:14:24.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:24.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:24.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-15T15:14:24.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:24.795+0000 7fa89ce11640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-15T15:14:27.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-15T15:14:27.677 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:27.711 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:27.711 INFO:tasks.ceph:role: osd.4 2024-04-15T15:14:27.711 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:14:27.712 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-15T15:14:27.791 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:27.791 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:27.792 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:28.285 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-04-15T15:14:28.287 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-04-15T15:14:28.288 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:28.395 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:28.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:28.500 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:28.500 INFO:tasks.ceph:role: osd.5 2024-04-15T15:14:28.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:14:28.500 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-15T15:14:28.582 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:28.582 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:28.583 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:28.583 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:28.583 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:29.052 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-04-15T15:14:29.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-04-15T15:14:29.054 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:29.147 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:29.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:29.249 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:29.249 INFO:tasks.ceph:role: osd.6 2024-04-15T15:14:29.249 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:14:29.250 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-15T15:14:29.322 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:29.322 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:29.322 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:29.322 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:29.323 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:29.824 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-04-15T15:14:29.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-04-15T15:14:29.826 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:29.962 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:29.995 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:30.065 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-15T15:14:30.065 INFO:tasks.ceph:role: osd.7 2024-04-15T15:14:30.065 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:14:30.065 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-15T15:14:30.142 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-15T15:14:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-15T15:14:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-15T15:14:30.142 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-15T15:14:30.143 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-15T15:14:30.666 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-04-15T15:14:30.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-04-15T15:14:30.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:30.804 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:30.842 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:30.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.942+0000 7f62f9448640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-15T15:14:30.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.942+0000 7f62f9448640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-15T15:14:30.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.942+0000 7f62f9448640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:30.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.942+0000 7f62f9448640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:30.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.942+0000 7f62f9448640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-15T15:14:30.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:30.943+0000 7f62f9448640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-15T15:14:33.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-15T15:14:33.830 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:33.925 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.923+0000 7fa937c9d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-15T15:14:33.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.923+0000 7fa937c9d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-15T15:14:33.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.923+0000 7fa937c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:33.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.923+0000 7fa937c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:33.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.923+0000 7fa937c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-15T15:14:33.926 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:33.924+0000 7fa937c9d640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-15T15:14:36.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-15T15:14:36.815 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:36.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.905+0000 7f355179a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-15T15:14:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.905+0000 7f355179a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-15T15:14:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.905+0000 7f355179a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.905+0000 7f355179a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.905+0000 7f355179a640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-15T15:14:36.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:36.906+0000 7f355179a640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-15T15:14:39.763 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-15T15:14:39.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-15T15:14:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-15T15:14:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:39.886 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-15T15:14:39.887 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:39.884+0000 7f3bdde1b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-15T15:14:41.274 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:41.277 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:14:42.733 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-15T15:14:42.766 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-15T15:14:42.766 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:42.766 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-15T15:14:42.800 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:42.800 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-15T15:14:42.872 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:42.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-15T15:14:42.943 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:42.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-15T15:14:43.013 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:43.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-15T15:14:43.082 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:43.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-15T15:14:43.151 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:43.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-15T15:14:43.220 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.220 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-15T15:14:43.253 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.253 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-15T15:14:43.325 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.325 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-15T15:14:43.395 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-15T15:14:43.465 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.465 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-15T15:14:43.535 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.535 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-15T15:14:43.603 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.603 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-15T15:14:43.672 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:43.673 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-15T15:14:43.688 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:43.688 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-15T15:14:43.744 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.745 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-15T15:14:43.762 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:43.762 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-15T15:14:43.817 INFO:tasks.ceph:Adding keys to all mons... 2024-04-15T15:14:43.818 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:14:43.820 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-15T15:14:43.861 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.y] 2024-04-15T15:14:43.861 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBFRB1m/2LdLRAASp0e6ytz/SwoCwpsA9IsMw== 2024-04-15T15:14:43.861 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.a] 2024-04-15T15:14:43.861 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBFRB1mKChoNRAAFnzGTfpU/J9EU9lWKg7E8w== 2024-04-15T15:14:43.861 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.c] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mvsphARAAlhGdDUn101JhENibKBJymQ== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBHRB1m5DZ6NBAAHq3NyjnLkmy2+12CH+2wAA== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBKRB1mOfrpMhAA1LErtVE7PABH7mYCQH5eaQ== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBNRB1mLOY6MRAACJAUnSukk7OtsJyIzyelxQ== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBQRB1m4RJ4LxAAgaJwWgQO58fok1arrpOUfA== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBFRB1mGCvQMRAAx+UFS92U9dNKJ59UmwU3zQ== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.b] 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1m8c2qBhAAB8LIa881VpeaEHNlYqR7RA== 2024-04-15T15:14:43.862 INFO:teuthology.orchestra.run.smithi070.stdout:[mds.d] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mF0tODhAACXE4CInpUuYPOaRVia6HsQ== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.4] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBWRB1mwgY7OBAAY8KIzT+psWeTYb2Gex0f5g== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.5] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBZRB1mzlofNxAAyFht43aiV+MwY9s81VTYMQ== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.6] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBcRB1m+2sKNhAADnej7b5JltPOhDlqH30lgw== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.7] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBfRB1m+ZzENBAAMhNOvxpkifAOpKvnQkO/zQ== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mV0TQEhAAVAq/vgY21Iz2b2lmIPnQPQ== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[client.2] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1ml+7MGBAAM+5SF+JwojvhE8UdWXFLLw== 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout:[client.1] 2024-04-15T15:14:43.863 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mUYdtHBAACC0J3a7+fz6P3ESJGyw1Ug== 2024-04-15T15:14:43.864 INFO:teuthology.orchestra.run.smithi070.stdout:[client.3] 2024-04-15T15:14:43.864 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mTpMUIBAA5Pe00R2+efb1TT5Q1vkT2g== 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.y] 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBFRB1m/2LdLRAASp0e6ytz/SwoCwpsA9IsMw== 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.a] 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBFRB1mKChoNRAAFnzGTfpU/J9EU9lWKg7E8w== 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.c] 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1mvsphARAAlhGdDUn101JhENibKBJymQ== 2024-04-15T15:14:43.886 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBHRB1m5DZ6NBAAHq3NyjnLkmy2+12CH+2wAA== 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBKRB1mOfrpMhAA1LErtVE7PABH7mYCQH5eaQ== 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBNRB1mLOY6MRAACJAUnSukk7OtsJyIzyelxQ== 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBQRB1m4RJ4LxAAgaJwWgQO58fok1arrpOUfA== 2024-04-15T15:14:43.887 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBFRB1mGCvQMRAAx+UFS92U9dNKJ59UmwU3zQ== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.b] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1m8c2qBhAAB8LIa881VpeaEHNlYqR7RA== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[mds.d] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1mF0tODhAACXE4CInpUuYPOaRVia6HsQ== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBWRB1mwgY7OBAAY8KIzT+psWeTYb2Gex0f5g== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBZRB1mzlofNxAAyFht43aiV+MwY9s81VTYMQ== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.6] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBcRB1m+2sKNhAADnej7b5JltPOhDlqH30lgw== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.7] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBfRB1m+ZzENBAAMhNOvxpkifAOpKvnQkO/zQ== 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-04-15T15:14:43.888 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1mV0TQEhAAVAq/vgY21Iz2b2lmIPnQPQ== 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout:[client.2] 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1ml+7MGBAAM+5SF+JwojvhE8UdWXFLLw== 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout:[client.1] 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1mUYdtHBAACC0J3a7+fz6P3ESJGyw1Ug== 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout:[client.3] 2024-04-15T15:14:43.889 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQBGRB1mTpMUIBAA5Pe00R2+efb1TT5Q1vkT2g== 2024-04-15T15:14:43.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:43.907 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:43.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:43.984 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.043 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.103 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.156 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.159 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.213 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.216 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.270 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.272 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:44.328 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-15T15:14:44.381 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.441 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-15T15:14:44.500 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.502 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.558 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.615 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.671 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-15T15:14:44.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.727 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.788 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.845 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.906 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-15T15:14:44.962 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-15T15:14:44.963 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-15T15:14:44.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:45.091 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-15T15:14:45.119 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-15T15:14:45.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:45.248 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-15T15:14:45.275 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-15T15:14:45.344 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-15T15:14:45.443 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-15T15:14:45.471 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:45.473 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-15T15:14:45.528 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-15T15:14:45.529 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-15T15:14:45.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-15T15:14:45.532 INFO:tasks.ceph.mon.a:Started 2024-04-15T15:14:45.532 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-15T15:14:45.532 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-15T15:14:45.570 INFO:tasks.ceph.mon.b:Started 2024-04-15T15:14:45.570 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-15T15:14:45.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-15T15:14:45.572 INFO:tasks.ceph.mon.c:Started 2024-04-15T15:14:45.573 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-15T15:14:45.573 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-15T15:14:45.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-15T15:14:45.575 INFO:tasks.ceph.mgr.y:Started 2024-04-15T15:14:45.575 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-15T15:14:45.576 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-15T15:14:45.578 INFO:tasks.ceph.mgr.x:Started 2024-04-15T15:14:45.578 INFO:tasks.ceph:Setting crush tunables to default 2024-04-15T15:14:45.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd crush tunables default 2024-04-15T15:14:45.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.679+0000 7f1cf9aa7640 1 Processor -- start 2024-04-15T15:14:45.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.679+0000 7f1cf9aa7640 1 -- start start 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf9aa7640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 0x7f1cf406bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf9aa7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf406c300 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf9aa7640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf40b67d0 con 0x7f1cf40b2950 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf9aa7640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf415faa0 con 0x7f1cf406b9e0 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1cf40ae4e0 0x7f1cf40ae8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59378/0 (socket says 172.21.15.70:59378) 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf3fff640 1 -- 172.21.15.70:0/1515933579 learned_addr learned my addr 172.21.15.70:0/1515933579 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 0x7f1cf406bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf37fe640 1 -- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 msgr2=0x7f1cf406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-15T15:14:45.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf37fe640 1 -- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 msgr2=0x7f1cf406bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-15T15:14:45.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 0x7f1cf406bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-15T15:14:45.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.680+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 0x7f1cf406bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-15T15:14:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.843+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1415362069 0 0) 0x7f1cf406c300 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.843+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ce0003520 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3536352646 0 0) 0x7f1ce0003520 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 msgr2=0x7f1cf406bdc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 --2- 172.21.15.70:0/1515933579 >> v2:172.21.15.96:3300/0 conn(0x7f1cf406b9e0 0x7f1cf406bdc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 >> v1:172.21.15.96:6789/0 conn(0x7f1cf40b2950 legacy=0x7f1cf40b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf415ee90 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f1cec0019f0 con 0x7f1cf40ae4e0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 >> v1:172.21.15.70:6789/0 conn(0x7f1cf40ae4e0 legacy=0x7f1cf40ae8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ce00039b0 0x7f1ce0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ce0004800 0x7f1ce0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1ce000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce000c0e0 con 0x7f1ce00051c0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce000c290 con 0x7f1ce00039b0 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce000c490 con 0x7f1ce0004800 2024-04-15T15:14:45.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf3fff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1ce000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ce00039b0 0x7f1ce0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cebfff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ce0004800 0x7f1ce0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.844+0000 7f1cf3fff640 1 -- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ce0004800 msgr2=0x7f1ce0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ce0004800 0x7f1ce0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 -- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ce00039b0 msgr2=0x7f1ce0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ce00039b0 0x7f1ce0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 -- 172.21.15.70:0/1515933579 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce0003520 con 0x7f1ce00051c0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ce00039b0 0x7f1ce0003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cebfff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ce0004800 0x7f1ce0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 -- 172.21.15.70:0/1515933579 reap_dead start 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.845+0000 7f1cf3fff640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1ce000b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00a850 tx=0x7f1ce0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bfedf0743ce0f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400b5e0 con 0x7f1ce00051c0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cec034650 con 0x7f1ce00051c0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf2ffd640 1 -- 172.21.15.70:0/1515933579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc001370 con 0x7f1ce00051c0 2024-04-15T15:14:45.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 msgr2=0x7f1ce000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1ce000b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00a850 tx=0x7f1ce0005830 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/1515933579 shutdown_connections 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/1515933579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1ce000b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/1515933579 >> 172.21.15.70:0/1515933579 conn(0x7f1cf4073260 msgr2=0x7f1cf409a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/1515933579 shutdown_connections 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.846+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/1515933579 wait complete. 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 Processor -- start 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 -- start start 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cf40b0860 0x7f1cf40b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1cf4151e40 0x7f1cf41522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf415f730 con 0x7f1ce00051c0 2024-04-15T15:14:45.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf415f450 con 0x7f1cf40b0860 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.848+0000 7f1cf9aa7640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf4159060 con 0x7f1cf4151e40 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50724/0 (socket says 172.21.15.70:50724) 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 -- 172.21.15.70:0/112710742 learned_addr learned my addr 172.21.15.70:0/112710742 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cebfff640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cf40b0860 0x7f1cf40b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf3fff640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1cf4151e40 0x7f1cf41522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 -- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1cf4151e40 msgr2=0x7f1cf41522c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1cf4151e40 0x7f1cf41522c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 -- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cf40b0860 msgr2=0x7f1cf40b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cf40b0860 0x7f1cf40b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 -- 172.21.15.70:0/112710742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf414f3b0 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf37fe640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40108c0 tx=0x7f1ce4010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce401a020 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce4010e50 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf414f630 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce400eb80 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.849+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf414fbf0 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.850+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1ce401d020 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.850+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1ce40147d0 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.850+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf40690f0 con 0x7f1ce00051c0 2024-04-15T15:14:45.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.852+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1ce4024020 con 0x7f1ce00051c0 2024-04-15T15:14:45.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.919+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f1cf4162dd0 con 0x7f1ce00051c0 2024-04-15T15:14:45.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.920+0000 7f1cf0ff9640 1 -- 172.21.15.70:0/112710742 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f1cf414f3b0 con 0x7f1ce00051c0 2024-04-15T15:14:45.922 INFO:teuthology.orchestra.run.smithi070.stderr:adjusted tunables profile to default 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 msgr2=0x7f1cf4142bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40108c0 tx=0x7f1ce4010cc0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 shutdown_connections 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1cf4151e40 0x7f1cf41522c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1cf40b0860 0x7f1cf40b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 --2- 172.21.15.70:0/112710742 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ce00051c0 0x7f1cf4142bb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 >> 172.21.15.70:0/112710742 conn(0x7f1cf4073260 msgr2=0x7f1cf4157f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:45.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 shutdown_connections 2024-04-15T15:14:45.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:45.921+0000 7f1cf9aa7640 1 -- 172.21.15.70:0/112710742 wait complete. 2024-04-15T15:14:45.934 INFO:tasks.ceph:check_enable_crimson: False 2024-04-15T15:14:45.934 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-15T15:14:45.934 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:45.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-15T15:14:45.962 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:45.962 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-15T15:14:46.031 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:46.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-15T15:14:46.098 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:14:46.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-15T15:14:46.167 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:46.167 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-15T15:14:46.195 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:46.195 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-15T15:14:46.264 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:46.265 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-15T15:14:46.305 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:46.303+0000 7fee5ea52040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:14:46.333 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:14:46.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-15T15:14:46.337 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:46.335+0000 7f4239641040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T15:14:46.363 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:46.361+0000 7fee5ea52040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:14:46.390 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:46.388+0000 7f4239641040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T15:14:46.391 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 03f8cc03-b81e-497c-91dd-ae79aba85fa5 0 2024-04-15T15:14:46.535 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.533+0000 7f62a34d3640 1 Processor -- start 2024-04-15T15:14:46.536 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.534+0000 7f62a34d3640 1 -- start start 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.534+0000 7f62a34d3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f629c0cee70 0x7f629c0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.534+0000 7f62a34d3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629c0cbee0 con 0x7f629c0cf650 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.534+0000 7f62a34d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f629c0cc1c0 con 0x7f629c15b520 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.534+0000 7f62a34d3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f629c0cc4a0 con 0x7f629c0cee70 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a126f640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f629c15b520 0x7f629c0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56918/0 (socket says 172.21.15.96:56918) 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a126f640 1 -- 172.21.15.96:0/1173480514 learned_addr learned my addr 172.21.15.96:0/1173480514 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a0a6e640 1 --2- 172.21.15.96:0/1173480514 >> v2:172.21.15.70:3300/0 conn(0x7f629c0cee70 0x7f629c0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.537 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1156235714 0 0) 0x7f629c0cc1c0 con 0x7f629c15b520 2024-04-15T15:14:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6288003520 con 0x7f629c15b520 2024-04-15T15:14:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 <== mon.1 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 966764094 0 0) 0x7f6288003520 con 0x7f629c15b520 2024-04-15T15:14:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a0a6e640 1 -- 172.21.15.96:0/1173480514 >> v1:172.21.15.96:6789/0 conn(0x7f629c15b520 legacy=0x7f629c0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a0a6e640 1 -- 172.21.15.96:0/1173480514 >> v1:172.21.15.70:6789/0 conn(0x7f629c0cf650 legacy=0x7f629c06eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.538 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.535+0000 7f62a0a6e640 1 -- 172.21.15.96:0/1173480514 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c0cc780 con 0x7f629c0cee70 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.536+0000 7f62a0a6e640 1 --2- 172.21.15.96:0/1173480514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0ccca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6290009720 tx=0x7f629002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b011a7bfeec9db server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.536+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290038070 con 0x7f629c0cee70 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.536+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6290031520 con 0x7f629c0cee70 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.536+0000 7f628ffff640 1 -- 172.21.15.96:0/1173480514 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6298001670 con 0x7f629c0cee70 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 -- 172.21.15.96:0/1173480514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 msgr2=0x7f629c0ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 --2- 172.21.15.96:0/1173480514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0ccca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6290009720 tx=0x7f629002b7c0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 -- 172.21.15.96:0/1173480514 shutdown_connections 2024-04-15T15:14:46.539 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 --2- 172.21.15.96:0/1173480514 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0ccca0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 -- 172.21.15.96:0/1173480514 >> 172.21.15.96:0/1173480514 conn(0x7f629c0b2e70 msgr2=0x7f629c0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 -- 172.21.15.96:0/1173480514 shutdown_connections 2024-04-15T15:14:46.540 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.537+0000 7f62a34d3640 1 -- 172.21.15.96:0/1173480514 wait complete. 2024-04-15T15:14:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.538+0000 7f62a34d3640 1 Processor -- start 2024-04-15T15:14:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.538+0000 7f62a34d3640 1 -- start start 2024-04-15T15:14:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.542 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 0x7f629c0cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f629c15b900 con 0x7f629c0cee70 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f629c0cfae0 con 0x7f629c0cf650 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f629c151ac0 con 0x7f629c15b520 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a1a70640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 0x7f629c0cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53186/0 (socket says 172.21.15.96:53186) 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a1a70640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 0x7f629c0cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56828/0 (socket says 172.21.15.96:56828) 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 -- 172.21.15.96:0/3395871609 learned_addr learned my addr 172.21.15.96:0/3395871609 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a126f640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 -- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 msgr2=0x7f629c0cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 0x7f629c0cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 -- 172.21.15.96:0/3395871609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 msgr2=0x7f629c0c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0c9590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.543 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 -- 172.21.15.96:0/3395871609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f629c0c9bc0 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a0a6e640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f629002b800 tx=0x7f62900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290043070 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f629003f870 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f629c0c9ea0 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.539+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6290038040 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.540+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f629c0cb1c0 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.540+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6290046070 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.540+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f6290048b20 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.541+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f629c0ca3c0 con 0x7f629c0cf650 2024-04-15T15:14:46.544 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.543+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f629003fc60 con 0x7f629c0cf650 2024-04-15T15:14:46.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.615+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd new", "uuid": "03f8cc03-b81e-497c-91dd-ae79aba85fa5", "id": 0} v 0) -- 0x7f629c0690f0 con 0x7f629c0cf650 2024-04-15T15:14:46.620 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.618+0000 7f628dffb640 1 -- 172.21.15.96:0/3395871609 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03f8cc03-b81e-497c-91dd-ae79aba85fa5", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f6290048450 con 0x7f629c0cf650 2024-04-15T15:14:46.620 INFO:teuthology.orchestra.run.smithi096.stdout:0 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 msgr2=0x7f629c151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f629002b800 tx=0x7f62900046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 shutdown_connections 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f629c15b520 0x7f629c0cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f629c0cf650 0x7f629c151580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.621 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 --2- 172.21.15.96:0/3395871609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f629c0cee70 0x7f629c0c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 >> 172.21.15.96:0/3395871609 conn(0x7f629c0b2e70 msgr2=0x7f629c074db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:46.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 shutdown_connections 2024-04-15T15:14:46.622 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.619+0000 7f62a34d3640 1 -- 172.21.15.96:0/3395871609 wait complete. 2024-04-15T15:14:46.631 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 38bc545f-bf09-4eb5-9ec9-4f66db112bfb 1 2024-04-15T15:14:46.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.732+0000 7f1b4b71c640 1 Processor -- start 2024-04-15T15:14:46.734 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.732+0000 7f1b4b71c640 1 -- start start 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 0x7f1b440b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4406d3a0 con 0x7f1b44155b40 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1b440b6ce0 con 0x7f1b4406c9f0 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b4b71c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1b440b6720 con 0x7f1b440d0470 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b49cb9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b48cb7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b48cb7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56836/0 (socket says 172.21.15.96:56836) 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b49cb9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53198/0 (socket says 172.21.15.96:53198) 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b48cb7640 1 -- 172.21.15.96:0/2990612255 learned_addr learned my addr 172.21.15.96:0/2990612255 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.733+0000 7f1b494b8640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 0x7f1b440b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.736 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 -- 172.21.15.96:0/2990612255 >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 msgr2=0x7f1b4406ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 -- 172.21.15.96:0/2990612255 >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 msgr2=0x7f1b440b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 0x7f1b440b5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 -- 172.21.15.96:0/2990612255 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b440b6a00 con 0x7f1b440d0470 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b494b8640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 0x7f1b440b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:46.737 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b49cb9640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:46.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/2990612255 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b44067720 tx=0x7f1b3802e0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78612e0706a6a6bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b37fff640 1 -- 172.21.15.96:0/2990612255 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3803b070 con 0x7f1b440d0470 2024-04-15T15:14:46.738 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b37fff640 1 -- 172.21.15.96:0/2990612255 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b38036440 con 0x7f1b440d0470 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.734+0000 7f1b37fff640 1 -- 172.21.15.96:0/2990612255 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b38036810 con 0x7f1b440d0470 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 -- 172.21.15.96:0/2990612255 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 msgr2=0x7f1b4415c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/2990612255 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1b44067720 tx=0x7f1b3802e0d0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 -- 172.21.15.96:0/2990612255 shutdown_connections 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.96:3300/0 conn(0x7f1b4406c9f0 0x7f1b4406ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/2990612255 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b4415c4a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/2990612255 >> v2:172.21.15.70:3300/0 conn(0x7f1b44155b40 0x7f1b440b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 -- 172.21.15.96:0/2990612255 >> 172.21.15.96:0/2990612255 conn(0x7f1b44072440 msgr2=0x7f1b44072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 -- 172.21.15.96:0/2990612255 shutdown_connections 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.735+0000 7f1b4b71c640 1 -- 172.21.15.96:0/2990612255 wait complete. 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.736+0000 7f1b4b71c640 1 Processor -- start 2024-04-15T15:14:46.739 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.736+0000 7f1b4b71c640 1 -- start start 2024-04-15T15:14:46.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 0x7f1b44154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.743 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 0x7f1b4415a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b440b6410 con 0x7f1b4406c9f0 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4415bf80 con 0x7f1b44155b40 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b4b71c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1b44155f20 con 0x7f1b440d0470 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b49cb9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 0x7f1b4415a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56844/0 (socket says 172.21.15.96:56844) 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 -- 172.21.15.96:0/4222077283 learned_addr learned my addr 172.21.15.96:0/4222077283 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:46.744 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b494b8640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 0x7f1b44154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 -- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 msgr2=0x7f1b4415a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 0x7f1b4415a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 -- 172.21.15.96:0/4222077283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 msgr2=0x7f1b44154fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 0x7f1b44154fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b48cb7640 1 -- 172.21.15.96:0/4222077283 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b44153200 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b49cb9640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 0x7f1b4415a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.737+0000 7f1b494b8640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 0x7f1b44154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b48cb7640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802e110 tx=0x7f1b380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3803b040 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b38046070 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b38034880 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b44156c20 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b44157180 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1b38034a20 con 0x7f1b440d0470 2024-04-15T15:14:46.745 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.738+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f1b3804bb20 con 0x7f1b440d0470 2024-04-15T15:14:46.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.739+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b440690f0 con 0x7f1b440d0470 2024-04-15T15:14:46.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.741+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1b44156c20 con 0x7f1b440d0470 2024-04-15T15:14:46.816 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.814+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd new", "uuid": "38bc545f-bf09-4eb5-9ec9-4f66db112bfb", "id": 1} v 0) -- 0x7f1b440cf080 con 0x7f1b440d0470 2024-04-15T15:14:46.819 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.817+0000 7f1b35ffb640 1 -- 172.21.15.96:0/4222077283 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38bc545f-bf09-4eb5-9ec9-4f66db112bfb", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f1b3804b460 con 0x7f1b440d0470 2024-04-15T15:14:46.819 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:14:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 msgr2=0x7f1b44158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1b3802e110 tx=0x7f1b380046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.820 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 shutdown_connections 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b44155b40 0x7f1b4415a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b440d0470 0x7f1b44158a00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 --2- 172.21.15.96:0/4222077283 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b4406c9f0 0x7f1b44154fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 >> 172.21.15.96:0/4222077283 conn(0x7f1b44072440 msgr2=0x7f1b44072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 shutdown_connections 2024-04-15T15:14:46.821 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.818+0000 7f1b4b71c640 1 -- 172.21.15.96:0/4222077283 wait complete. 2024-04-15T15:14:46.831 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 3be06c65-ca7d-4627-96ce-ecfd69b5d1c8 2 2024-04-15T15:14:46.931 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:46.929+0000 7fee5ea52040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:14:46.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.934+0000 7f0f0a5bd640 1 Processor -- start 2024-04-15T15:14:46.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.934+0000 7f0f0a5bd640 1 -- start start 2024-04-15T15:14:46.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 0x7f0f040d1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.939 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0f040d1840 0x7f0f0415f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0f0415ba70 con 0x7f0f040d08a0 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0f040ce5b0 con 0x7f0f040d1840 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f0a5bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0f040ce890 con 0x7f0f0415b690 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0f040d1840 0x7f0f0415f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56858/0 (socket says 172.21.15.96:56858) 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0f040d1840 0x7f0f0415f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53210/0 (socket says 172.21.15.96:53210) 2024-04-15T15:14:46.940 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f03fff640 1 -- 172.21.15.96:0/4188338490 learned_addr learned my addr 172.21.15.96:0/4188338490 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f037fe640 1 --2- 172.21.15.96:0/4188338490 >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 0x7f0f040d1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 -- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 msgr2=0x7f0f040b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 -- 172.21.15.96:0/4188338490 >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 msgr2=0x7f0f040d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 --2- 172.21.15.96:0/4188338490 >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 0x7f0f040d1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f08b5a640 1 -- 172.21.15.96:0/4188338490 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f040ceb70 con 0x7f0f040d1840 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f03fff640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.935+0000 7f0f037fe640 1 --2- 172.21.15.96:0/4188338490 >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 0x7f0f040d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.936+0000 7f0f08b5a640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d1840 0x7f0f0415f1d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f04072cf0 tx=0x7f0efc02be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=748a902897d424fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.936+0000 7f0f02ffd640 1 -- 172.21.15.96:0/4188338490 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc039070 con 0x7f0f040d1840 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.936+0000 7f0f02ffd640 1 -- 172.21.15.96:0/4188338490 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0efc02cbd0 con 0x7f0f040d1840 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.936+0000 7f0f02ffd640 1 -- 172.21.15.96:0/4188338490 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0efc034440 con 0x7f0f040d1840 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d1840 msgr2=0x7f0f0415f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d1840 0x7f0f0415f1d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0f04072cf0 tx=0x7f0efc02be20 comp rx=0 tx=0).stop 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/4188338490 shutdown_connections 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d1840 0x7f0f0415f1d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/4188338490 >> v2:172.21.15.70:3300/0 conn(0x7f0f040d08a0 0x7f0f040d1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/4188338490 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f0415b690 0x7f0f040b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/4188338490 >> 172.21.15.96:0/4188338490 conn(0x7f0f0409a9d0 msgr2=0x7f0f040b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/4188338490 shutdown_connections 2024-04-15T15:14:46.942 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.937+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/4188338490 wait complete. 2024-04-15T15:14:46.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.939+0000 7f0f0a5bd640 1 Processor -- start 2024-04-15T15:14:46.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.939+0000 7f0f0a5bd640 1 -- start start 2024-04-15T15:14:46.944 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f0415b690 0x7f0f0415de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f0415f710 con 0x7f0f0415b690 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0f040d1010 con 0x7f0f040d08a0 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f0a5bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0415f890 con 0x7f0f040d1840 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f03fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53226/0 (socket says 172.21.15.96:53226) 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f03fff640 1 -- 172.21.15.96:0/2380751482 learned_addr learned my addr 172.21.15.96:0/2380751482 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f037fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56860/0 (socket says 172.21.15.96:56860) 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.940+0000 7f0f08b5a640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f0415b690 0x7f0f0415de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 -- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 msgr2=0x7f0f0414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.945 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 -- 172.21.15.96:0/2380751482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f0415b690 msgr2=0x7f0f0415de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f0415b690 0x7f0f0415de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 -- 172.21.15.96:0/2380751482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f0414d440 con 0x7f0f040d08a0 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f037fe640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f03fff640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e9e0 tx=0x7f0ef400eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4017070 con 0x7f0f040d08a0 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef400c9d0 con 0x7f0f040d08a0 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef400cd80 con 0x7f0f040d08a0 2024-04-15T15:14:46.946 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f0414d720 con 0x7f0f040d08a0 2024-04-15T15:14:46.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f0414dce0 con 0x7f0f040d08a0 2024-04-15T15:14:46.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0ef401a070 con 0x7f0f040d08a0 2024-04-15T15:14:46.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.941+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f0ef401c8b0 con 0x7f0f040d08a0 2024-04-15T15:14:46.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.942+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f040690f0 con 0x7f0f040d08a0 2024-04-15T15:14:46.947 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:46.944+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0ef401c450 con 0x7f0f040d08a0 2024-04-15T15:14:46.952 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:46.950+0000 7f4239641040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T15:14:46.990 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:46.988+0000 7fee5ea52040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:14:47.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.004+0000 7f4239641040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T15:14:47.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.017+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3be06c65-ca7d-4627-96ce-ecfd69b5d1c8", "id": 2} v 0) -- 0x7f0f04070150 con 0x7f0f040d08a0 2024-04-15T15:14:47.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.020+0000 7f0f00ff9640 1 -- 172.21.15.96:0/2380751482 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3be06c65-ca7d-4627-96ce-ecfd69b5d1c8", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f0ef401cb50 con 0x7f0f040d08a0 2024-04-15T15:14:47.022 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 msgr2=0x7f0f041595f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e9e0 tx=0x7f0ef400eea0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 shutdown_connections 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f0415b690 0x7f0f0415de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f040d1840 0x7f0f0414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 --2- 172.21.15.96:0/2380751482 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f040d08a0 0x7f0f041595f0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.021+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 >> 172.21.15.96:0/2380751482 conn(0x7f0f0409a9d0 msgr2=0x7f0f04070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.022+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 shutdown_connections 2024-04-15T15:14:47.024 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.022+0000 7f0f0a5bd640 1 -- 172.21.15.96:0/2380751482 wait complete. 2024-04-15T15:14:47.034 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 8710d76b-5689-4f90-ae4a-b9d135325ead 3 2024-04-15T15:14:47.037 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.036+0000 7fee5ea52040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:14:47.052 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.050+0000 7f4239641040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T15:14:47.096 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.094+0000 7fee5ea52040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:14:47.108 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.106+0000 7f4239641040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T15:14:47.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.136+0000 7f04bad8d640 1 Processor -- start 2024-04-15T15:14:47.138 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.136+0000 7f04bad8d640 1 -- start start 2024-04-15T15:14:47.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04bad8d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.140 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04bad8d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 0x7f04b40b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04bad8d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04b415ba70 con 0x7f04b415b690 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04bad8d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f04b40d0d50 con 0x7f04b40d08a0 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04bad8d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f04b40ce5b0 con 0x7f04b40d1a10 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56876/0 (socket says 172.21.15.96:56876) 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 -- 172.21.15.96:0/1703729587 learned_addr learned my addr 172.21.15.96:0/1703729587 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b932a640 1 --2- 172.21.15.96:0/1703729587 >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 0x7f04b40b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 -- 172.21.15.96:0/1703729587 >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 msgr2=0x7f04b40b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 --2- 172.21.15.96:0/1703729587 >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 0x7f04b40b1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 -- 172.21.15.96:0/1703729587 >> v1:172.21.15.70:6789/0 conn(0x7f04b415b690 legacy=0x7f04b40d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.141 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b3fff640 1 -- 172.21.15.96:0/1703729587 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b40ce890 con 0x7f04b40d1a10 2024-04-15T15:14:47.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.137+0000 7f04b932a640 1 --2- 172.21.15.96:0/1703729587 >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 0x7f04b40b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:47.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.138+0000 7f04b3fff640 1 --2- 172.21.15.96:0/1703729587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ffef546175cc34 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.138+0000 7f04b37fe640 1 -- 172.21.15.96:0/1703729587 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0034070 con 0x7f04b40d1a10 2024-04-15T15:14:47.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.138+0000 7f04b37fe640 1 -- 172.21.15.96:0/1703729587 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a0030840 con 0x7f04b40d1a10 2024-04-15T15:14:47.142 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.138+0000 7f04b37fe640 1 -- 172.21.15.96:0/1703729587 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a0030c10 con 0x7f04b40d1a10 2024-04-15T15:14:47.142 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.138+0000 7fee5ea52040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:14:47.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 -- 172.21.15.96:0/1703729587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 msgr2=0x7f04b40b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 --2- 172.21.15.96:0/1703729587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a002b980 comp rx=0 tx=0).stop 2024-04-15T15:14:47.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 -- 172.21.15.96:0/1703729587 shutdown_connections 2024-04-15T15:14:47.145 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 --2- 172.21.15.96:0/1703729587 >> v2:172.21.15.70:3300/0 conn(0x7f04b40d08a0 0x7f04b40b1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 --2- 172.21.15.96:0/1703729587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d1a10 0x7f04b40b6ff0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 -- 172.21.15.96:0/1703729587 >> 172.21.15.96:0/1703729587 conn(0x7f04b409a9d0 msgr2=0x7f04b40b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 -- 172.21.15.96:0/1703729587 shutdown_connections 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.139+0000 7f04bad8d640 1 -- 172.21.15.96:0/1703729587 wait complete. 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.140+0000 7f04bad8d640 1 Processor -- start 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.140+0000 7f04bad8d640 1 -- start start 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04b415b690 0x7f04b4148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04b40b1ba0 con 0x7f04b415b690 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04b40d1010 con 0x7f04b40d1a10 2024-04-15T15:14:47.146 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04bad8d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f04b40b1d20 con 0x7f04b40d08a0 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56886/0 (socket says 172.21.15.96:56886) 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b3fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53242/0 (socket says 172.21.15.96:53242) 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b932a640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04b415b690 0x7f04b4148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 -- 172.21.15.96:0/2066507783 learned_addr learned my addr 172.21.15.96:0/2066507783 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 -- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 msgr2=0x7f04b409b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 -- 172.21.15.96:0/2066507783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04b415b690 msgr2=0x7f04b4148130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04b415b690 0x7f04b4148130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.147 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b8b29640 1 -- 172.21.15.96:0/2066507783 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b4146140 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.141+0000 7f04b3fff640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b8b29640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c3c0 tx=0x7f04a800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a80052d0 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a8005490 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b4146420 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8007260 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b41469e0 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f04a8007400 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.142+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f04a8011590 con 0x7f04b40d08a0 2024-04-15T15:14:47.148 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.143+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b40690f0 con 0x7f04b40d08a0 2024-04-15T15:14:47.149 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.145+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f04b415ba70 con 0x7f04b40d08a0 2024-04-15T15:14:47.152 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.150+0000 7f4239641040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T15:14:47.219 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.217+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd new", "uuid": "8710d76b-5689-4f90-ae4a-b9d135325ead", "id": 3} v 0) -- 0x7f04b4146c20 con 0x7f04b40d08a0 2024-04-15T15:14:47.222 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.220+0000 7f04b17fa640 1 -- 172.21.15.96:0/2066507783 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8710d76b-5689-4f90-ae4a-b9d135325ead", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f04a8003a00 con 0x7f04b40d08a0 2024-04-15T15:14:47.222 INFO:teuthology.orchestra.run.smithi096.stdout:3 2024-04-15T15:14:47.223 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 msgr2=0x7f04b4145980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f04a800c3c0 tx=0x7f04a800c880 comp rx=0 tx=0).stop 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 shutdown_connections 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04b415b690 0x7f04b4148130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04b40d1a10 0x7f04b409b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 --2- 172.21.15.96:0/2066507783 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04b40d08a0 0x7f04b4145980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 >> 172.21.15.96:0/2066507783 conn(0x7f04b409a9d0 msgr2=0x7f04b406f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 shutdown_connections 2024-04-15T15:14:47.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.221+0000 7f04bad8d640 1 -- 172.21.15.96:0/2066507783 wait complete. 2024-04-15T15:14:47.234 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 2a0965cf-b047-4bae-8af5-03d83051fb3d 4 2024-04-15T15:14:47.265 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.262+0000 7fee5ea52040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:14:47.273 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.272+0000 7f4239641040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-15T15:14:47.338 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.336+0000 7fee5ea52040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:47.345 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.343+0000 7f4239641040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 Processor -- start 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 -- start start 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f258415e6f0 0x7f258406c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258406f220 con 0x7f258406ecb0 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f258406f500 con 0x7f258406cf30 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.366+0000 7f258a972640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f25840b45d0 con 0x7f258415e6f0 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f25837fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f258406cf30 0x7f258406e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.96:56926/0 (socket says 172.21.15.96:56926) 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f2583fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f258415e6f0 0x7f258406c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.369 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f25837fe640 1 -- 172.21.15.96:0/2130273334 learned_addr learned my addr 172.21.15.96:0/2130273334 (peer_addr_for_me v1:172.21.15.96:0/0) 2024-04-15T15:14:47.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f2583fff640 1 -- 172.21.15.96:0/2130273334 >> v1:172.21.15.96:6789/0 conn(0x7f258406cf30 legacy=0x7f258406e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f2583fff640 1 -- 172.21.15.96:0/2130273334 >> v1:172.21.15.70:6789/0 conn(0x7f258406ecb0 legacy=0x7f25840b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.370 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.367+0000 7f2583fff640 1 -- 172.21.15.96:0/2130273334 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25840b48b0 con 0x7f258415e6f0 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f2583fff640 1 --2- 172.21.15.96:0/2130273334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258415e6f0 0x7f258406c9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25740097a0 tx=0x7f257402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1892062dd3876db server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f2582ffd640 1 -- 172.21.15.96:0/2130273334 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574034070 con 0x7f258415e6f0 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f2582ffd640 1 -- 172.21.15.96:0/2130273334 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2574030610 con 0x7f258415e6f0 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f2582ffd640 1 -- 172.21.15.96:0/2130273334 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25740309e0 con 0x7f258415e6f0 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 -- 172.21.15.96:0/2130273334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258415e6f0 msgr2=0x7f258406c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 --2- 172.21.15.96:0/2130273334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258415e6f0 0x7f258406c9f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f25740097a0 tx=0x7f257402b960 comp rx=0 tx=0).stop 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 -- 172.21.15.96:0/2130273334 shutdown_connections 2024-04-15T15:14:47.371 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 --2- 172.21.15.96:0/2130273334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258415e6f0 0x7f258406c9f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 -- 172.21.15.96:0/2130273334 >> 172.21.15.96:0/2130273334 conn(0x7f2584072440 msgr2=0x7f2584072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 -- 172.21.15.96:0/2130273334 shutdown_connections 2024-04-15T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.368+0000 7f258a972640 1 -- 172.21.15.96:0/2130273334 wait complete. 2024-04-15T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.370+0000 7f258a972640 1 Processor -- start 2024-04-15T15:14:47.372 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.371+0000 7f258a972640 1 -- start start 2024-04-15T15:14:47.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 0x7f25841575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 0x7f2584157ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.375 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f258415ec50 con 0x7f258415e6f0 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f258415ead0 con 0x7f258406cf30 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f258a972640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f25840b54e0 con 0x7f258406ecb0 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f25837fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 0x7f25841575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53272/0 (socket says 172.21.15.96:53272) 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 -- 172.21.15.96:0/3412134821 learned_addr learned my addr 172.21.15.96:0/3412134821 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2588f0f640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 0x7f2584157ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 -- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 msgr2=0x7f25841575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 0x7f25841575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 -- 172.21.15.96:0/3412134821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 msgr2=0x7f2584157ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 0x7f2584157ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 -- 172.21.15.96:0/3412134821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2584158210 con 0x7f258406cf30 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2588f0f640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 0x7f2584157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:47.376 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f25837fe640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 0x7f25841575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2583fff640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f257402bf70 tx=0x7f25740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574034040 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257403acc0 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.372+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2574044630 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.373+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25841584f0 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.373+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2584158ab0 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.373+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f257404b060 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.373+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f257404d960 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.373+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25840690f0 con 0x7f258406cf30 2024-04-15T15:14:47.377 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.376+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2574052020 con 0x7f258406cf30 2024-04-15T15:14:47.389 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.387+0000 7fee5ea52040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:14:47.392 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.391+0000 7f4239641040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T15:14:47.443 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.441+0000 7fee5ea52040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:14:47.446 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.445+0000 7f4239641040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T15:14:47.450 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.448+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2a0965cf-b047-4bae-8af5-03d83051fb3d", "id": 4} v 0) -- 0x7f25840b14c0 con 0x7f258406cf30 2024-04-15T15:14:47.453 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.451+0000 7f2580ff9640 1 -- 172.21.15.96:0/3412134821 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2a0965cf-b047-4bae-8af5-03d83051fb3d", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f257403f070 con 0x7f258406cf30 2024-04-15T15:14:47.453 INFO:teuthology.orchestra.run.smithi096.stdout:4 2024-04-15T15:14:47.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 msgr2=0x7f258414c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f257402bf70 tx=0x7f25740046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.454 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 shutdown_connections 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f258415e6f0 0x7f2584157ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f258406ecb0 0x7f25841575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 --2- 172.21.15.96:0/3412134821 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f258406cf30 0x7f258414c5f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 >> 172.21.15.96:0/3412134821 conn(0x7f2584072440 msgr2=0x7f25840b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 shutdown_connections 2024-04-15T15:14:47.455 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.452+0000 7f258a972640 1 -- 172.21.15.96:0/3412134821 wait complete. 2024-04-15T15:14:47.464 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new df277b1e-2fe5-48a8-907b-364d770dbe65 5 2024-04-15T15:14:47.494 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.492+0000 7fee5ea52040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:14:47.495 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.493+0000 7f4239641040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T15:14:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.565+0000 7f1603515640 1 Processor -- start 2024-04-15T15:14:47.567 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.565+0000 7f1603515640 1 -- start start 2024-04-15T15:14:47.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1603515640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f15fc15ed00 0x7f15fc070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1603515640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f15fc0cf600 0x7f15fc0cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.571 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1603515640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15fc0ae440 con 0x7f15fc06ef70 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1603515640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc0ae720 con 0x7f15fc15ed00 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1603515640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f15fc0aea00 con 0x7f15fc0cf600 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f15fc0cf600 0x7f15fc0cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f15fc0cf600 0x7f15fc0cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53288/0 (socket says 172.21.15.96:53288) 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 -- 172.21.15.96:0/3259644533 learned_addr learned my addr 172.21.15.96:0/3259644533 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f16012b1640 1 --2- 172.21.15.96:0/3259644533 >> v2:172.21.15.70:3300/0 conn(0x7f15fc15ed00 0x7f15fc070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 -- 172.21.15.96:0/3259644533 >> v2:172.21.15.70:3300/0 conn(0x7f15fc15ed00 msgr2=0x7f15fc070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:47.572 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 --2- 172.21.15.96:0/3259644533 >> v2:172.21.15.70:3300/0 conn(0x7f15fc15ed00 0x7f15fc070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 -- 172.21.15.96:0/3259644533 >> v1:172.21.15.70:6789/0 conn(0x7f15fc06ef70 legacy=0x7f15fc06f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.566+0000 7f1600ab0640 1 -- 172.21.15.96:0/3259644533 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc0aece0 con 0x7f15fc0cf600 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.567+0000 7f1600ab0640 1 --2- 172.21.15.96:0/3259644533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0cfa90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f15fc09ade0 tx=0x7f15ec02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d998ba8a0ddd10 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.567+0000 7f15f3fff640 1 -- 172.21.15.96:0/3259644533 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec034070 con 0x7f15fc0cf600 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.567+0000 7f15f3fff640 1 -- 172.21.15.96:0/3259644533 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15ec030860 con 0x7f15fc0cf600 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.567+0000 7f15f3fff640 1 -- 172.21.15.96:0/3259644533 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ec030c30 con 0x7f15fc0cf600 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 -- 172.21.15.96:0/3259644533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 msgr2=0x7f15fc0cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 --2- 172.21.15.96:0/3259644533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0cfa90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f15fc09ade0 tx=0x7f15ec02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.573 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 -- 172.21.15.96:0/3259644533 shutdown_connections 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 --2- 172.21.15.96:0/3259644533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0cfa90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 --2- 172.21.15.96:0/3259644533 >> v2:172.21.15.70:3300/0 conn(0x7f15fc15ed00 0x7f15fc070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 -- 172.21.15.96:0/3259644533 >> 172.21.15.96:0/3259644533 conn(0x7f15fc09a9d0 msgr2=0x7f15fc0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 -- 172.21.15.96:0/3259644533 shutdown_connections 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.568+0000 7f1603515640 1 -- 172.21.15.96:0/3259644533 wait complete. 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.569+0000 7f1603515640 1 Processor -- start 2024-04-15T15:14:47.574 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.569+0000 7f1603515640 1 -- start start 2024-04-15T15:14:47.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15fc06ef70 0x7f15fc14c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 0x7f15fc0c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.576 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc070e60 con 0x7f15fc06ef70 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f15fc0701a0 con 0x7f15fc0cf600 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f15fc070fe0 con 0x7f15fc15ed00 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1600ab0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1600ab0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53296/0 (socket says 172.21.15.96:53296) 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1600ab0640 1 -- 172.21.15.96:0/568361158 learned_addr learned my addr 172.21.15.96:0/568361158 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 0x7f15fc0c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f16012b1640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15fc06ef70 0x7f15fc14c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 -- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 msgr2=0x7f15fc0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 -- 172.21.15.96:0/568361158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15fc06ef70 msgr2=0x7f15fc14c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15fc06ef70 0x7f15fc14c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.577 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 -- 172.21.15.96:0/568361158 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15fc0c91d0 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1600ab0640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1601ab2640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 0x7f15fc0c8ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f15f4010a00 tx=0x7f15f4010ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f4003b70 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f4003d30 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15fc0c94b0 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.570+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f400ebb0 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.571+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f15fc0c9a70 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.571+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f15f40050d0 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.571+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f15f4016e00 con 0x7f15fc15ed00 2024-04-15T15:14:47.578 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.572+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15fc0690f0 con 0x7f15fc15ed00 2024-04-15T15:14:47.579 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.574+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f15f4016420 con 0x7f15fc15ed00 2024-04-15T15:14:47.648 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.646+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd new", "uuid": "df277b1e-2fe5-48a8-907b-364d770dbe65", "id": 5} v 0) -- 0x7f15fc06fdb0 con 0x7f15fc15ed00 2024-04-15T15:14:47.651 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.649+0000 7f15f1ffb640 1 -- 172.21.15.96:0/568361158 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "df277b1e-2fe5-48a8-907b-364d770dbe65", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f15f400ed50 con 0x7f15fc15ed00 2024-04-15T15:14:47.651 INFO:teuthology.orchestra.run.smithi096.stdout:5 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 msgr2=0x7f15fc0c8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 0x7f15fc0c8ba0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f15f4010a00 tx=0x7f15f4010ec0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 shutdown_connections 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f15fc15ed00 0x7f15fc0c8ba0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f15fc0cf600 0x7f15fc0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.653 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 --2- 172.21.15.96:0/568361158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f15fc06ef70 0x7f15fc14c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 >> 172.21.15.96:0/568361158 conn(0x7f15fc09a9d0 msgr2=0x7f15fc0cfa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 shutdown_connections 2024-04-15T15:14:47.654 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.650+0000 7f1603515640 1 -- 172.21.15.96:0/568361158 wait complete. 2024-04-15T15:14:47.663 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new bfc1e2f6-658b-4c4b-9871-16b54fb2b075 6 2024-04-15T15:14:47.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.803+0000 7f6a40cc1640 1 Processor -- start 2024-04-15T15:14:47.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 -- start start 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 0x7f6a3c09a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a3c0d0350 con 0x7f6a3c09af10 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6a3c0b5b00 con 0x7f6a3c09b650 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a40cc1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c0ce100 con 0x7f6a3c15c8d0 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a33fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 0x7f6a3c09a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56918/0 (socket says 172.21.15.96:56918) 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.804+0000 7f6a3a59c640 1 -- 172.21.15.96:0/548057674 learned_addr learned my addr 172.21.15.96:0/548057674 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a3a59c640 1 -- 172.21.15.96:0/548057674 >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 msgr2=0x7f6a3c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a3a59c640 1 --2- 172.21.15.96:0/548057674 >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 0x7f6a3c09a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.808 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a3a59c640 1 -- 172.21.15.96:0/548057674 >> v1:172.21.15.70:6789/0 conn(0x7f6a3c09af10 legacy=0x7f6a3c0d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a3a59c640 1 -- 172.21.15.96:0/548057674 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c0b54b0 con 0x7f6a3c15c8d0 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a33fff640 1 --2- 172.21.15.96:0/548057674 >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 0x7f6a3c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.805+0000 7f6a3a59c640 1 --2- 172.21.15.96:0/548057674 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009870 tx=0x7f6a2c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f415b5cc2a51bd79 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a39d9b640 1 -- 172.21.15.96:0/548057674 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c035020 con 0x7f6a3c15c8d0 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a39d9b640 1 -- 172.21.15.96:0/548057674 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c031640 con 0x7f6a3c15c8d0 2024-04-15T15:14:47.809 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a39d9b640 1 -- 172.21.15.96:0/548057674 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c02d040 con 0x7f6a3c15c8d0 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 -- 172.21.15.96:0/548057674 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 msgr2=0x7f6a3c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/548057674 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009870 tx=0x7f6a2c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 -- 172.21.15.96:0/548057674 shutdown_connections 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/548057674 >> v2:172.21.15.96:3300/0 conn(0x7f6a3c09b650 0x7f6a3c09a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/548057674 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c15c8d0 0x7f6a3c0cda70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 -- 172.21.15.96:0/548057674 >> 172.21.15.96:0/548057674 conn(0x7f6a3c0b3fb0 msgr2=0x7f6a3c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 -- 172.21.15.96:0/548057674 shutdown_connections 2024-04-15T15:14:47.812 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.806+0000 7f6a40cc1640 1 -- 172.21.15.96:0/548057674 wait complete. 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 Processor -- start 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 -- start start 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c09b650 0x7f6a3c14d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c157700 0x7f6a3c157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c0b7120 con 0x7f6a3c09b650 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6a3c0b6fa0 con 0x7f6a3c09af10 2024-04-15T15:14:47.813 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.808+0000 7f6a40cc1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6a3c0b72a0 con 0x7f6a3c157700 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53316/0 (socket says 172.21.15.96:53316) 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 -- 172.21.15.96:0/1270981477 learned_addr learned my addr 172.21.15.96:0/1270981477 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3ad9d640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c157700 0x7f6a3c157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a33fff640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c09b650 0x7f6a3c14d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 -- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c157700 msgr2=0x7f6a3c157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c157700 0x7f6a3c157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 -- 172.21.15.96:0/1270981477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c09b650 msgr2=0x7f6a3c14d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c09b650 0x7f6a3c14d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 -- 172.21.15.96:0/1270981477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a3c156790 con 0x7f6a3c09af10 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a3a59c640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000c00 tx=0x7f6a2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c040070 con 0x7f6a3c09af10 2024-04-15T15:14:47.814 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a2c0399f0 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a3c1582c0 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c035040 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a3c0c7860 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6a2c04b060 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.809+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f6a2c03aa10 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.810+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a3c0690f0 con 0x7f6a3c09af10 2024-04-15T15:14:47.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.812+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f6a2c063e50 con 0x7f6a3c09af10 2024-04-15T15:14:47.879 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:47.877+0000 7f4239641040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:14:47.888 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.886+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bfc1e2f6-658b-4c4b-9871-16b54fb2b075", "id": 6} v 0) -- 0x7f6a3c0cecb0 con 0x7f6a3c09af10 2024-04-15T15:14:47.891 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:47.889+0000 7fee5ea52040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T15:14:47.894 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.892+0000 7f6a32ffd640 1 -- 172.21.15.96:0/1270981477 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfc1e2f6-658b-4c4b-9871-16b54fb2b075", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f6a2c052070 con 0x7f6a3c09af10 2024-04-15T15:14:47.894 INFO:teuthology.orchestra.run.smithi096.stdout:6 2024-04-15T15:14:47.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 msgr2=0x7f6a3c14c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:47.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000c00 tx=0x7f6a2c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 shutdown_connections 2024-04-15T15:14:47.895 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6a3c157700 0x7f6a3c157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6a3c09b650 0x7f6a3c14d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 --2- 172.21.15.96:0/1270981477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6a3c09af10 0x7f6a3c14c970 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:47.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 >> 172.21.15.96:0/1270981477 conn(0x7f6a3c0b3fb0 msgr2=0x7f6a3c06ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:47.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 shutdown_connections 2024-04-15T15:14:47.896 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:47.893+0000 7f6a40cc1640 1 -- 172.21.15.96:0/1270981477 wait complete. 2024-04-15T15:14:47.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 2adb9d7e-7c61-48e4-8c49-c3b4929d2954 7 2024-04-15T15:14:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.008+0000 7f5f3a8b1640 1 Processor -- start 2024-04-15T15:14:48.010 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.008+0000 7f5f3a8b1640 1 -- start start 2024-04-15T15:14:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5f3415b520 0x7f5f340ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.012 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5f340cbee0 con 0x7f5f3415b520 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5f340cc1c0 con 0x7f5f340cee70 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f3a8b1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340cc4a0 con 0x7f5f340cf650 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f337fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56928/0 (socket says 172.21.15.96:56928) 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f337fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53330/0 (socket says 172.21.15.96:53330) 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 -- 172.21.15.96:0/596179435 learned_addr learned my addr 172.21.15.96:0/596179435 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:48.013 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f33fff640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.70:3300/0 conn(0x7f5f3415b520 0x7f5f340ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 -- 172.21.15.96:0/596179435 >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 msgr2=0x7f5f340ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 -- 172.21.15.96:0/596179435 >> v2:172.21.15.70:3300/0 conn(0x7f5f3415b520 msgr2=0x7f5f340ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.70:3300/0 conn(0x7f5f3415b520 0x7f5f340ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f38e4e640 1 -- 172.21.15.96:0/596179435 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340cc780 con 0x7f5f340cf650 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.009+0000 7f5f337fe640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.010+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/596179435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5f34072320 tx=0x7f5f2c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7db04d318cb1df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.010+0000 7f5f32ffd640 1 -- 172.21.15.96:0/596179435 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02ca70 con 0x7f5f340cf650 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.010+0000 7f5f32ffd640 1 -- 172.21.15.96:0/596179435 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02cc10 con 0x7f5f340cf650 2024-04-15T15:14:48.014 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.010+0000 7f5f32ffd640 1 -- 172.21.15.96:0/596179435 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c0343f0 con 0x7f5f340cf650 2024-04-15T15:14:48.015 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.011+0000 7f4239641040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/596179435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 msgr2=0x7f5f340cfae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/596179435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5f34072320 tx=0x7f5f2c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/596179435 shutdown_connections 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/596179435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f340cf650 0x7f5f340cfae0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.96:3300/0 conn(0x7f5f340cee70 0x7f5f340ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/596179435 >> v2:172.21.15.70:3300/0 conn(0x7f5f3415b520 0x7f5f340ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.018 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/596179435 >> 172.21.15.96:0/596179435 conn(0x7f5f340b2e70 msgr2=0x7f5f340b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/596179435 shutdown_connections 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.011+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/596179435 wait complete. 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.012+0000 7f5f3a8b1640 1 Processor -- start 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.012+0000 7f5f3a8b1640 1 -- start start 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 0x7f5f341f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 0x7f5f340c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.019 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5f3415b900 con 0x7f5f340cf650 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5f340cf3a0 con 0x7f5f340cee70 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f3a8b1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5f340c5970 con 0x7f5f3415b520 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 0x7f5f341f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f38e4e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f38e4e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.96:56936/0 (socket says 172.21.15.96:56936) 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f33fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 0x7f5f341f5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.96:53332/0 (socket says 172.21.15.96:53332) 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f38e4e640 1 -- 172.21.15.96:0/2820982953 learned_addr learned my addr 172.21.15.96:0/2820982953 (peer_addr_for_me v2:172.21.15.96:0/0) 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.013+0000 7f5f337fe640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 0x7f5f340c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 -- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 msgr2=0x7f5f341f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 0x7f5f341f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 -- 172.21.15.96:0/2820982953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 msgr2=0x7f5f340c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 0x7f5f340c8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.020 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 -- 172.21.15.96:0/2820982953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f340c5be0 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f337fe640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 0x7f5f340c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f38e4e640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036040 tx=0x7f5f2c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c043020 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f2c03e450 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f340c5ec0 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c009510 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f340c6480 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5f2c046020 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.014+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f5f2c048a60 con 0x7f5f3415b520 2024-04-15T15:14:48.021 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.015+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f340690f0 con 0x7f5f3415b520 2024-04-15T15:14:48.022 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.017+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f5f2c04d030 con 0x7f5f3415b520 2024-04-15T15:14:48.036 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.035+0000 7fee5ea52040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T15:14:48.092 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.089+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2adb9d7e-7c61-48e4-8c49-c3b4929d2954", "id": 7} v 0) -- 0x7f5f340c6650 con 0x7f5f3415b520 2024-04-15T15:14:48.100 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f30ff9640 1 -- 172.21.15.96:0/2820982953 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2adb9d7e-7c61-48e4-8c49-c3b4929d2954", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f5f2c02ca80 con 0x7f5f3415b520 2024-04-15T15:14:48.100 INFO:teuthology.orchestra.run.smithi096.stdout:7 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 msgr2=0x7f5f340c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c036040 tx=0x7f5f2c004a40 comp rx=0 tx=0).stop 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 shutdown_connections 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5f3415b520 0x7f5f340c88b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5f340cf650 0x7f5f340c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 --2- 172.21.15.96:0/2820982953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5f340cee70 0x7f5f341f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.098+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 >> 172.21.15.96:0/2820982953 conn(0x7f5f340b2e70 msgr2=0x7f5f340b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.101 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.099+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 shutdown_connections 2024-04-15T15:14:48.102 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:14:48.099+0000 7f5f3a8b1640 1 -- 172.21.15.96:0/2820982953 wait complete. 2024-04-15T15:14:48.111 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-15T15:14:48.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-15T15:14:48.114 INFO:tasks.ceph.osd.0:Started 2024-04-15T15:14:48.114 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-15T15:14:48.114 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-15T15:14:48.117 INFO:tasks.ceph.osd.1:Started 2024-04-15T15:14:48.117 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-15T15:14:48.117 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-15T15:14:48.119 INFO:tasks.ceph.osd.2:Started 2024-04-15T15:14:48.119 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-15T15:14:48.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-15T15:14:48.121 INFO:tasks.ceph.osd.3:Started 2024-04-15T15:14:48.121 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-15T15:14:48.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-15T15:14:48.124 INFO:tasks.ceph.osd.4:Started 2024-04-15T15:14:48.124 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-15T15:14:48.124 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-15T15:14:48.126 INFO:tasks.ceph.osd.5:Started 2024-04-15T15:14:48.126 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-15T15:14:48.126 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-15T15:14:48.128 INFO:tasks.ceph.osd.6:Started 2024-04-15T15:14:48.129 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-15T15:14:48.129 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-15T15:14:48.130 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.127+0000 7f4239641040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:14:48.133 INFO:tasks.ceph.osd.7:Started 2024-04-15T15:14:48.134 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:48.167 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.165+0000 7fee5ea52040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T15:14:48.204 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.203+0000 7f4239641040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:14:48.233 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.232+0000 7fee5ea52040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T15:14:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.255+0000 7fcb5a4fb640 1 Processor -- start 2024-04-15T15:14:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.255+0000 7fcb5a4fb640 1 -- start start 2024-04-15T15:14:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb5a4fb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb540d06b0 0x7fcb540d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb5a4fb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 0x7fcb540cf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb5a4fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb5406b9e0 con 0x7fcb54153670 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb5a4fb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcb540d1060 con 0x7fcb540d06b0 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb5a4fb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcb540d23b0 con 0x7fcb540cf160 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb540d06b0 0x7fcb540d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcb540d06b0 0x7fcb540d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50766/0 (socket says 172.21.15.70:50766) 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 -- 172.21.15.70:0/627377779 learned_addr learned my addr 172.21.15.70:0/627377779 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb58a98640 1 --2- 172.21.15.70:0/627377779 >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 0x7fcb540cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 -- 172.21.15.70:0/627377779 >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 msgr2=0x7fcb540cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 --2- 172.21.15.70:0/627377779 >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 0x7fcb540cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 -- 172.21.15.70:0/627377779 >> v1:172.21.15.70:6789/0 conn(0x7fcb54153670 legacy=0x7fcb540d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb537fe640 1 -- 172.21.15.70:0/627377779 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb5406bc80 con 0x7fcb540d06b0 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.256+0000 7fcb58a98640 1 --2- 172.21.15.70:0/627377779 >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 0x7fcb540cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:14:48.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.257+0000 7fcb537fe640 1 --2- 172.21.15.70:0/627377779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb540d0b20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb540cefc0 tx=0x7fcb4802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd95fd1bcde36ea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.257+0000 7fcb52ffd640 1 -- 172.21.15.70:0/627377779 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48035020 con 0x7fcb540d06b0 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.257+0000 7fcb52ffd640 1 -- 172.21.15.70:0/627377779 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb48031870 con 0x7fcb540d06b0 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.257+0000 7fcb52ffd640 1 -- 172.21.15.70:0/627377779 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48031c40 con 0x7fcb540d06b0 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/627377779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 msgr2=0x7fcb540d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/627377779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb540d0b20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcb540cefc0 tx=0x7fcb4802b910 comp rx=0 tx=0).stop 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/627377779 shutdown_connections 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/627377779 >> v2:172.21.15.96:3300/0 conn(0x7fcb540cf160 0x7fcb540cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/627377779 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb540d0b20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/627377779 >> 172.21.15.70:0/627377779 conn(0x7fcb5409a9d0 msgr2=0x7fcb540d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/627377779 shutdown_connections 2024-04-15T15:14:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.258+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/627377779 wait complete. 2024-04-15T15:14:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.259+0000 7fcb5a4fb640 1 Processor -- start 2024-04-15T15:14:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.259+0000 7fcb5a4fb640 1 -- start start 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb540cf160 0x7fcb54160830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb54153670 0x7fcb5415dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb54153a50 con 0x7fcb540d06b0 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb5406c240 con 0x7fcb540cf160 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb5a4fb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcb5415d9c0 con 0x7fcb54153670 2024-04-15T15:14:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50768/0 (socket says 172.21.15.70:50768) 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 -- 172.21.15.70:0/2327215208 learned_addr learned my addr 172.21.15.70:0/2327215208 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb53fff640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb540cf160 0x7fcb54160830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb58a98640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb54153670 0x7fcb5415dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 -- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb54153670 msgr2=0x7fcb5415dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb54153670 0x7fcb5415dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 -- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb540cf160 msgr2=0x7fcb54160830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb540cf160 0x7fcb54160830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 -- 172.21.15.70:0/2327215208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb540cab90 con 0x7fcb540d06b0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.260+0000 7fcb537fe640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb48000c00 tx=0x7fcb480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb48035040 con 0x7fcb540d06b0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb48030d00 con 0x7fcb540d06b0 2024-04-15T15:14:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb480456a0 con 0x7fcb540d06b0 2024-04-15T15:14:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb541f2930 con 0x7fcb540d06b0 2024-04-15T15:14:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb541f2ef0 con 0x7fcb540d06b0 2024-04-15T15:14:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcb48038020 con 0x7fcb540d06b0 2024-04-15T15:14:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.261+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fcb4804e3a0 con 0x7fcb540d06b0 2024-04-15T15:14:48.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.262+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb541f30f0 con 0x7fcb540d06b0 2024-04-15T15:14:48.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.266+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fcb541f30f0 con 0x7fcb540d06b0 2024-04-15T15:14:48.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.326+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcb540690f0 con 0x7fcb540d06b0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.327+0000 7fcb50ff9640 1 -- 172.21.15.70:0/2327215208 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fcb4803a420 con 0x7fcb540d06b0 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:14:48.329 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":10,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:14:48.092043+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 msgr2=0x7fcb5414e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:48.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcb48000c00 tx=0x7fcb480046c0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 shutdown_connections 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb54153670 0x7fcb5415dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb540d06b0 0x7fcb5414e020 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 --2- 172.21.15.70:0/2327215208 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb540cf160 0x7fcb54160830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 >> 172.21.15.70:0/2327215208 conn(0x7fcb5409a9d0 msgr2=0x7fcb540b2470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 shutdown_connections 2024-04-15T15:14:48.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:14:48.328+0000 7fcb5a4fb640 1 -- 172.21.15.70:0/2327215208 wait complete. 2024-04-15T15:14:48.331 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.328+0000 7f4239641040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:48.341 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-15T15:14:48.341 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-15T15:14:48.346 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.344+0000 7fee5ea52040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:48.428 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.426+0000 7f4239641040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:14:48.440 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.438+0000 7fee5ea52040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T15:14:48.584 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.583+0000 7f4239641040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:48.597 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.595+0000 7fee5ea52040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T15:14:48.626 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.624+0000 7f4239641040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:48.639 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.637+0000 7fee5ea52040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T15:14:48.717 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-15T15:14:48.715+0000 7f1d4260e640 -1 Falling back to public interface 2024-04-15T15:14:48.728 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-04-15T15:14:48.727+0000 7fd37c51d640 -1 Falling back to public interface 2024-04-15T15:14:48.737 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-04-15T15:14:48.736+0000 7f281559a640 -1 Falling back to public interface 2024-04-15T15:14:48.743 INFO:tasks.ceph.osd.6.smithi096.stderr:2024-04-15T15:14:48.742+0000 7ff916817640 -1 Falling back to public interface 2024-04-15T15:14:48.747 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-15T15:14:48.745+0000 7f633e213640 -1 Falling back to public interface 2024-04-15T15:14:48.751 INFO:tasks.ceph.osd.7.smithi096.stderr:2024-04-15T15:14:48.750+0000 7f3512e20640 -1 Falling back to public interface 2024-04-15T15:14:48.756 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-15T15:14:48.754+0000 7fe0e80cc640 -1 Falling back to public interface 2024-04-15T15:14:48.761 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-15T15:14:48.759+0000 7f2f3b49d640 -1 Falling back to public interface 2024-04-15T15:14:48.785 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:48.783+0000 7f4239641040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:14:48.800 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:48.799+0000 7fee5ea52040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T15:14:49.148 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:49.241 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.239+0000 7f8d4f42c640 1 Processor -- start 2024-04-15T15:14:49.242 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.239+0000 7f8d4f42c640 1 -- start start 2024-04-15T15:14:49.245 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4f42c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8d480b11b0 0x7f8d480d3290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.245 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4f42c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d480d4590 0x7f8d480d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.245 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4f42c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d480b4530 con 0x7f8d4815ead0 2024-04-15T15:14:49.245 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4f42c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8d480b4e20 con 0x7f8d480d4590 2024-04-15T15:14:49.245 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4f42c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8d4815eeb0 con 0x7f8d480b11b0 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4d9c9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d480d4590 0x7f8d480d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4d1c8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8d4815ead0 0x7f8d480cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:59410/0 (socket says 172.21.15.70:59410) 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4d9c9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d480d4590 0x7f8d480d4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50776/0 (socket says 172.21.15.70:50776) 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4d1c8640 1 -- 172.21.15.70:0/2458398671 learned_addr learned my addr 172.21.15.70:0/2458398671 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.240+0000 7f8d4c9c7640 1 --2- 172.21.15.70:0/2458398671 >> v2:172.21.15.96:3300/0 conn(0x7f8d480b11b0 0x7f8d480d3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d4d9c9640 1 -- 172.21.15.70:0/2458398671 >> v2:172.21.15.96:3300/0 conn(0x7f8d480b11b0 msgr2=0x7f8d480d3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d4d9c9640 1 --2- 172.21.15.70:0/2458398671 >> v2:172.21.15.96:3300/0 conn(0x7f8d480b11b0 0x7f8d480d3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d4d9c9640 1 -- 172.21.15.70:0/2458398671 >> v1:172.21.15.70:6789/0 conn(0x7f8d4815ead0 legacy=0x7f8d480cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d3ffff640 1 -- 172.21.15.70:0/2458398671 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786279732 0 0) 0x7f8d480b4530 con 0x7f8d4815ead0 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d4d9c9640 1 -- 172.21.15.70:0/2458398671 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d480d04a0 con 0x7f8d480d4590 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.241+0000 7f8d4d9c9640 1 --2- 172.21.15.70:0/2458398671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480d4590 0x7f8d480d4a20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d480d5250 tx=0x7f8d4002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd8665ebcf35673 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d3ffff640 1 -- 172.21.15.70:0/2458398671 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40039070 con 0x7f8d480d4590 2024-04-15T15:14:49.246 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d3ffff640 1 -- 172.21.15.70:0/2458398671 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d4002cc80 con 0x7f8d480d4590 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d3ffff640 1 -- 172.21.15.70:0/2458398671 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d40034660 con 0x7f8d480d4590 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2458398671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480d4590 msgr2=0x7f8d480d4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2458398671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480d4590 0x7f8d480d4a20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d480d5250 tx=0x7f8d4002bed0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2458398671 shutdown_connections 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2458398671 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480d4590 0x7f8d480d4a20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2458398671 >> v2:172.21.15.96:3300/0 conn(0x7f8d480b11b0 0x7f8d480d3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2458398671 >> 172.21.15.70:0/2458398671 conn(0x7f8d4809d9c0 msgr2=0x7f8d4808bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2458398671 shutdown_connections 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.242+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2458398671 wait complete. 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.243+0000 7f8d4f42c640 1 Processor -- start 2024-04-15T15:14:49.247 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.243+0000 7f8d4f42c640 1 -- start start 2024-04-15T15:14:49.252 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.252 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d480d4590 0x7f8d480cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.252 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d4815ead0 0x7f8d480cdde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:49.253 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d480b15b0 con 0x7f8d480b11b0 2024-04-15T15:14:49.253 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d480b4ab0 con 0x7f8d480d4590 2024-04-15T15:14:49.253 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4f42c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8d480b52b0 con 0x7f8d4815ead0 2024-04-15T15:14:49.253 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50790/0 (socket says 172.21.15.70:50790) 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 -- 172.21.15.70:0/2949753153 learned_addr learned my addr 172.21.15.70:0/2949753153 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4c9c7640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d480d4590 0x7f8d480cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d9c9640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d4815ead0 0x7f8d480cdde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 -- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d4815ead0 msgr2=0x7f8d480cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.254 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d4815ead0 0x7f8d480cdde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 -- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d480d4590 msgr2=0x7f8d480cd8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d480d4590 0x7f8d480cd8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.244+0000 7f8d4d1c8640 1 -- 172.21.15.70:0/2949753153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d481f4220 con 0x7f8d480b11b0 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d4d1c8640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800c430 tx=0x7f8d3800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d380074e0 con 0x7f8d480b11b0 2024-04-15T15:14:49.255 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d380047e0 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d38004bd0 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d481f47a0 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8d38011070 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d481f4d60 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.245+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f8d380123b0 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.246+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d481f4f60 con 0x7f8d480b11b0 2024-04-15T15:14:49.256 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.248+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f8d481f4f60 con 0x7f8d480b11b0 2024-04-15T15:14:49.311 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.309+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8d48075b20 con 0x7f8d480b11b0 2024-04-15T15:14:49.311 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.309+0000 7f8d3dffb640 1 -- 172.21.15.70:0/2949753153 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f8d38004dc0 con 0x7f8d480b11b0 2024-04-15T15:14:49.311 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-15T15:14:49.312 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":10,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:14:48.092043+0000","last_up_change":"0.000000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:49.312 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 msgr2=0x7f8d4815c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f8d3800c430 tx=0x7f8d3800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 shutdown_connections 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d4815ead0 0x7f8d480cdde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d480d4590 0x7f8d480cd8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 --2- 172.21.15.70:0/2949753153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d480b11b0 0x7f8d4815c230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 >> 172.21.15.70:0/2949753153 conn(0x7f8d4809d9c0 msgr2=0x7f8d4806d880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 shutdown_connections 2024-04-15T15:14:49.313 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:49.310+0000 7f8d4f42c640 1 -- 172.21.15.70:0/2949753153 wait complete. 2024-04-15T15:14:49.321 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-15T15:14:49.614 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:49.612+0000 7f4239641040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:14:49.640 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:49.639+0000 7fee5ea52040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T15:14:49.893 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:49.891+0000 7f4239641040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:14:49.919 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:49.917+0000 7fee5ea52040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T15:14:49.945 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:49.942+0000 7f4239641040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:14:49.969 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:49.968+0000 7fee5ea52040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T15:14:50.058 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:50.057+0000 7f4239641040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:14:50.087 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-04-15T15:14:50.086+0000 7fee5ea52040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T15:14:51.073 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:51.071+0000 7f41da351640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-15T15:14:51.073 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:14:51.072+0000 7f41da351640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-15T15:14:52.188 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-15T15:14:52.186+0000 7f1d4260e640 -1 osd.1 0 log_to_monitors true 2024-04-15T15:14:52.189 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-04-15T15:14:52.188+0000 7fd37c51d640 -1 osd.4 0 log_to_monitors true 2024-04-15T15:14:52.234 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-04-15T15:14:52.233+0000 7f281559a640 -1 osd.5 0 log_to_monitors true 2024-04-15T15:14:52.236 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-15T15:14:52.234+0000 7fe0e80cc640 -1 osd.0 0 log_to_monitors true 2024-04-15T15:14:52.236 INFO:tasks.ceph.osd.6.smithi096.stderr:2024-04-15T15:14:52.236+0000 7ff916817640 -1 osd.6 0 log_to_monitors true 2024-04-15T15:14:52.246 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-15T15:14:52.245+0000 7f633e213640 -1 osd.2 0 log_to_monitors true 2024-04-15T15:14:52.259 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-15T15:14:52.257+0000 7f2f3b49d640 -1 osd.3 0 log_to_monitors true 2024-04-15T15:14:52.269 INFO:tasks.ceph.osd.7.smithi096.stderr:2024-04-15T15:14:52.268+0000 7f3512e20640 -1 osd.7 0 log_to_monitors true 2024-04-15T15:14:55.447 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-15T15:14:55.445+0000 7f1d3dfd3640 -1 osd.1 0 waiting for initial osdmap 2024-04-15T15:14:55.456 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-15T15:14:55.453+0000 7f6339bc0640 -1 osd.2 0 waiting for initial osdmap 2024-04-15T15:14:55.463 INFO:tasks.ceph.osd.1.smithi070.stderr:2024-04-15T15:14:55.462+0000 7f1d35df3640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:55.465 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-04-15T15:14:55.464+0000 7f63319e0640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:55.550 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-15T15:14:55.548+0000 7fe0e4062640 -1 osd.0 0 waiting for initial osdmap 2024-04-15T15:14:55.558 INFO:tasks.ceph.osd.0.smithi070.stderr:2024-04-15T15:14:55.557+0000 7fe0dbe82640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:55.564 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-15T15:14:55.563+0000 7f2f37c47640 -1 osd.3 0 waiting for initial osdmap 2024-04-15T15:14:55.580 INFO:tasks.ceph.osd.3.smithi070.stderr:2024-04-15T15:14:55.579+0000 7f2f2f253640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:56.090 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-04-15T15:14:56.089+0000 7fd3784b3640 -1 osd.4 0 waiting for initial osdmap 2024-04-15T15:14:56.102 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-04-15T15:14:56.101+0000 7fd3702d3640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:56.128 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:14:56.136 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-04-15T15:14:56.135+0000 7f2811530640 -1 osd.5 0 waiting for initial osdmap 2024-04-15T15:14:56.152 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-04-15T15:14:56.151+0000 7f2809350640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:56.174 INFO:tasks.ceph.osd.6.smithi096.stderr:2024-04-15T15:14:56.173+0000 7ff9120ed640 -1 osd.6 0 waiting for initial osdmap 2024-04-15T15:14:56.190 INFO:tasks.ceph.osd.6.smithi096.stderr:2024-04-15T15:14:56.189+0000 7ff909f0d640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:56.227 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.225+0000 7f3599a9a640 1 Processor -- start 2024-04-15T15:14:56.227 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.225+0000 7f3599a9a640 1 -- start start 2024-04-15T15:14:56.228 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f3599a9a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f359415ead0 0x7f35940d43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f3599a9a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f35940d48e0 0x7f35940cf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f3599a9a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35940d4d70 con 0x7f35940cf750 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f3599a9a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f35940b4530 con 0x7f359415ead0 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f3599a9a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f359415eeb0 con 0x7f35940d48e0 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f35937fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f359415ead0 0x7f35940d43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f35937fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f359415ead0 0x7f35940d43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47004/0 (socket says 172.21.15.70:47004) 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f35937fe640 1 -- 172.21.15.70:0/994490736 learned_addr learned my addr 172.21.15.70:0/994490736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.226+0000 7f358bfff640 1 --2- 172.21.15.70:0/994490736 >> v2:172.21.15.96:3300/0 conn(0x7f35940d48e0 0x7f35940cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.227+0000 7f35937fe640 1 -- 172.21.15.70:0/994490736 >> v2:172.21.15.96:3300/0 conn(0x7f35940d48e0 msgr2=0x7f35940cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.227+0000 7f35937fe640 1 --2- 172.21.15.70:0/994490736 >> v2:172.21.15.96:3300/0 conn(0x7f35940d48e0 0x7f35940cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.227+0000 7f35937fe640 1 -- 172.21.15.70:0/994490736 >> v1:172.21.15.70:6789/0 conn(0x7f35940cf750 legacy=0x7f35940d0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.229 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.227+0000 7f35937fe640 1 -- 172.21.15.70:0/994490736 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35940b3e30 con 0x7f359415ead0 2024-04-15T15:14:56.230 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.227+0000 7f35937fe640 1 --2- 172.21.15.70:0/994490736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940d43a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f359406c3a0 tx=0x7f358402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af880a165a7f2f3f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.230 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3592ffd640 1 -- 172.21.15.70:0/994490736 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584035020 con 0x7f359415ead0 2024-04-15T15:14:56.230 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3592ffd640 1 -- 172.21.15.70:0/994490736 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3584031870 con 0x7f359415ead0 2024-04-15T15:14:56.230 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3592ffd640 1 -- 172.21.15.70:0/994490736 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3584031c40 con 0x7f359415ead0 2024-04-15T15:14:56.231 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 -- 172.21.15.70:0/994490736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 msgr2=0x7f35940d43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.231 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 --2- 172.21.15.70:0/994490736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940d43a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f359406c3a0 tx=0x7f358402b910 comp rx=0 tx=0).stop 2024-04-15T15:14:56.231 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 -- 172.21.15.70:0/994490736 shutdown_connections 2024-04-15T15:14:56.231 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 --2- 172.21.15.70:0/994490736 >> v2:172.21.15.96:3300/0 conn(0x7f35940d48e0 0x7f35940cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.231 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 --2- 172.21.15.70:0/994490736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940d43a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.232 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 -- 172.21.15.70:0/994490736 >> 172.21.15.70:0/994490736 conn(0x7f359409d9c0 msgr2=0x7f359408bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:56.232 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 -- 172.21.15.70:0/994490736 shutdown_connections 2024-04-15T15:14:56.232 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.229+0000 7f3599a9a640 1 -- 172.21.15.70:0/994490736 wait complete. 2024-04-15T15:14:56.232 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.231+0000 7f3599a9a640 1 Processor -- start 2024-04-15T15:14:56.232 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.231+0000 7f3599a9a640 1 -- start start 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.231+0000 7f3599a9a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f35940cf750 0x7f35940ca030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35940d48e0 0x7f359415cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35940d0470 con 0x7f359415ead0 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f35940d0190 con 0x7f35940d48e0 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f35940cfbc0 con 0x7f35940cf750 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47006/0 (socket says 172.21.15.70:47006) 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f35937fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f35940cf750 0x7f35940ca030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.235 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 -- 172.21.15.70:0/1549869336 learned_addr learned my addr 172.21.15.70:0/1549869336 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f358bfff640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35940d48e0 0x7f359415cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 -- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f35940cf750 msgr2=0x7f35940ca030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f35940cf750 0x7f35940ca030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 -- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35940d48e0 msgr2=0x7f359415cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35940d48e0 0x7f359415cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 -- 172.21.15.70:0/1549869336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3594160f40 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3593fff640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c760 tx=0x7f358c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c003b90 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f358c003d30 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c00aa30 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35940c9270 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.232+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35940c9b10 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.233+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35940c9660 con 0x7f359415ead0 2024-04-15T15:14:56.236 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.234+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 4) ==== 90984+0+0 (secure 0 0 0) 0x7f358c00abd0 con 0x7f359415ead0 2024-04-15T15:14:56.237 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.234+0000 7f3590ff9640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 0x7f357406ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:14:56.237 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.234+0000 7f35937fe640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 0x7f357406ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:14:56.237 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.234+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 4191+0+0 (secure 0 0 0) 0x7f358c08b190 con 0x7f359415ead0 2024-04-15T15:14:56.237 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.235+0000 7f35937fe640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 0x7f357406ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3584036000 tx=0x7f358403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:14:56.241 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.239+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f358c059640 con 0x7f359415ead0 2024-04-15T15:14:56.364 INFO:tasks.ceph.osd.7.smithi096.stderr:2024-04-15T15:14:56.363+0000 7f350e6f8640 -1 osd.7 0 waiting for initial osdmap 2024-04-15T15:14:56.376 INFO:tasks.ceph.osd.7.smithi096.stderr:2024-04-15T15:14:56.375+0000 7f3506518640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T15:14:56.421 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.419+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3594075b20 con 0x7f359415ead0 2024-04-15T15:14:56.421 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.420+0000 7f3590ff9640 1 -- 172.21.15.70:0/1549869336 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) ==== 74+0+10375 (secure 0 0 0) 0x7f358c05d940 con 0x7f359415ead0 2024-04-15T15:14:56.422 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-15T15:14:56.422 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":13,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:14:56.217716+0000","last_up_change":"2024-04-15T15:14:56.217716+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 msgr2=0x7f357406ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 0x7f357406ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3584036000 tx=0x7f358403c040 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 msgr2=0x7f35940cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f358c00c760 tx=0x7f358c00cc20 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 shutdown_connections 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f357406c860 0x7f357406ed20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f359415ead0 0x7f35940cd6e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f35940d48e0 0x7f359415cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 --2- 172.21.15.70:0/1549869336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f35940cf750 0x7f35940ca030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:14:56.424 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 >> 172.21.15.70:0/1549869336 conn(0x7f359409d9c0 msgr2=0x7f35940d09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:14:56.425 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 shutdown_connections 2024-04-15T15:14:56.425 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:14:56.422+0000 7f3599a9a640 1 -- 172.21.15.70:0/1549869336 wait complete. 2024-04-15T15:14:56.433 DEBUG:teuthology.misc:7 of 8 OSDs are up 2024-04-15T15:15:03.241 DEBUG:teuthology.orchestra.run.smithi070:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:03.343 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.341+0000 7f2b67532640 1 Processor -- start 2024-04-15T15:15:03.343 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.341+0000 7f2b67532640 1 -- start start 2024-04-15T15:15:03.344 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b67532640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2b6015ea40 0x7f2b600d0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b67532640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2b600af3e0 0x7f2b60072940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b67532640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b600afaa0 con 0x7f2b600d0b90 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b67532640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2b600af850 con 0x7f2b600af3e0 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b67532640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2b60071b80 con 0x7f2b6015ea40 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b65acf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2b600af3e0 0x7f2b60072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b65acf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2b600af3e0 0x7f2b60072940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47902/0 (socket says 172.21.15.70:47902) 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.342+0000 7f2b65acf640 1 -- 172.21.15.70:0/3958178736 learned_addr learned my addr 172.21.15.70:0/3958178736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b652ce640 1 --2- 172.21.15.70:0/3958178736 >> v2:172.21.15.96:3300/0 conn(0x7f2b6015ea40 0x7f2b600d0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b65acf640 1 -- 172.21.15.70:0/3958178736 >> v2:172.21.15.96:3300/0 conn(0x7f2b6015ea40 msgr2=0x7f2b600d0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b65acf640 1 --2- 172.21.15.70:0/3958178736 >> v2:172.21.15.96:3300/0 conn(0x7f2b6015ea40 0x7f2b600d0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b65acf640 1 -- 172.21.15.70:0/3958178736 >> v1:172.21.15.96:6789/0 conn(0x7f2b600d0b90 legacy=0x7f2b600aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.345 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b65acf640 1 -- 172.21.15.70:0/3958178736 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b60071e60 con 0x7f2b600af3e0 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.343+0000 7f2b65acf640 1 --2- 172.21.15.70:0/3958178736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b60072940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009e50 tx=0x7f2b5802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286845f509f14491 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b57fff640 1 -- 172.21.15.70:0/3958178736 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5802cb20 con 0x7f2b600af3e0 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b57fff640 1 -- 172.21.15.70:0/3958178736 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5802ccc0 con 0x7f2b600af3e0 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b57fff640 1 -- 172.21.15.70:0/3958178736 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b58034610 con 0x7f2b600af3e0 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 -- 172.21.15.70:0/3958178736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 msgr2=0x7f2b60072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.347 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 --2- 172.21.15.70:0/3958178736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b60072940 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2b58009e50 tx=0x7f2b5802bf80 comp rx=0 tx=0).stop 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 -- 172.21.15.70:0/3958178736 shutdown_connections 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 --2- 172.21.15.70:0/3958178736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b60072940 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 --2- 172.21.15.70:0/3958178736 >> v2:172.21.15.96:3300/0 conn(0x7f2b6015ea40 0x7f2b600d0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 -- 172.21.15.70:0/3958178736 >> 172.21.15.70:0/3958178736 conn(0x7f2b6006be00 msgr2=0x7f2b6006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 -- 172.21.15.70:0/3958178736 shutdown_connections 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.344+0000 7f2b67532640 1 -- 172.21.15.70:0/3958178736 wait complete. 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 Processor -- start 2024-04-15T15:15:03.348 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 -- start start 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2b600d0b90 0x7f2b60099390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 0x7f2b6015a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2b600b3160 con 0x7f2b600af3e0 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2b60072420 con 0x7f2b6015ea40 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b67532640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2b60072e80 con 0x7f2b600d0b90 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b652ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.351 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.346+0000 7f2b652ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47914/0 (socket says 172.21.15.70:47914) 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 -- 172.21.15.70:0/3947152435 learned_addr learned my addr 172.21.15.70:0/3947152435 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b64acd640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2b600d0b90 0x7f2b60099390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b65acf640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 0x7f2b6015a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 -- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2b600d0b90 msgr2=0x7f2b60099390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2b600d0b90 0x7f2b60099390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 -- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 msgr2=0x7f2b6015a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 0x7f2b6015a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 -- 172.21.15.70:0/3947152435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b6015b0c0 con 0x7f2b600af3e0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b65acf640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 0x7f2b6015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b652ce640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00d5d0 tx=0x7f2b5c00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0125f0 con 0x7f2b600af3e0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b5c019070 con 0x7f2b600af3e0 2024-04-15T15:15:03.352 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b6015b3a0 con 0x7f2b600af3e0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b5c0129f0 con 0x7f2b600af3e0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.347+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b6015b960 con 0x7f2b600af3e0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.348+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b60075b20 con 0x7f2b600af3e0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.349+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2b5c012b90 con 0x7f2b600af3e0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.349+0000 7f2b55ffb640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 0x7f2b4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.349+0000 7f2b64acd640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 0x7f2b4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.353 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.349+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f2b5c08bc50 con 0x7f2b600af3e0 2024-04-15T15:15:03.354 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.350+0000 7f2b64acd640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 0x7f2b4406edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b600b4280 tx=0x7f2b50009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.356 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.354+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2b5c091020 con 0x7f2b600af3e0 2024-04-15T15:15:03.531 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.529+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b601443f0 con 0x7f2b600af3e0 2024-04-15T15:15:03.532 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.530+0000 7f2b55ffb640 1 -- 172.21.15.70:0/3947152435 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12824 (secure 0 0 0) 0x7f2b5c05a3d0 con 0x7f2b600af3e0 2024-04-15T15:15:03.532 INFO:teuthology.misc.health.smithi070.stdout: 2024-04-15T15:15:03.532 INFO:teuthology.misc.health.smithi070.stdout:{"epoch":17,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:00.252437+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 msgr2=0x7f2b4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 0x7f2b4406edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2b600b4280 tx=0x7f2b50009040 comp rx=0 tx=0).stop 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 msgr2=0x7f2b6015c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c00d5d0 tx=0x7f2b5c00da90 comp rx=0 tx=0).stop 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 shutdown_connections 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2b6015ea40 0x7f2b6015a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2b600d0b90 0x7f2b60099390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.534 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2b4406c8f0 0x7f2b4406edb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.535 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 --2- 172.21.15.70:0/3947152435 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2b600af3e0 0x7f2b6015c160 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.535 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 >> 172.21.15.70:0/3947152435 conn(0x7f2b6006be00 msgr2=0x7f2b6009d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.535 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 shutdown_connections 2024-04-15T15:15:03.535 INFO:teuthology.misc.health.smithi070.stderr:2024-04-15T15:15:03.532+0000 7f2b67532640 1 -- 172.21.15.70:0/3947152435 wait complete. 2024-04-15T15:15:03.544 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-15T15:15:03.544 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-15T15:15:03.544 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:15:03.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:15:03.546 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:15:03.546 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:15:03.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:15:03.548 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:15:03.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:15:03.549 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:15:03.552 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:15:03.552 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:15:03.552 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:15:03.554 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:15:03.554 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-15T15:15:03.555 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:03.625 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-15T15:15:03.626 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-15T15:15:03.626 INFO:tasks.ceph.mds.c.smithi070.stderr:2024-04-15T15:15:03.624+0000 7f5b2b627640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-15T15:15:03.640 INFO:tasks.ceph.mds.d.smithi096.stdout:starting mds.d at 2024-04-15T15:15:03.641 INFO:tasks.ceph.mds.b.smithi096.stdout:starting mds.b at 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.662+0000 7fbe96e42640 1 Processor -- start 2024-04-15T15:15:03.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.662+0000 7fbe96e42640 1 -- start start 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe96e42640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe90153670 0x7fbe900d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe96e42640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 0x7fbe900cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe96e42640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe9006b9e0 con 0x7fbe900d21e0 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe96e42640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9006bc80 con 0x7fbe90153670 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe96e42640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbe9006bf60 con 0x7fbe900d0dc0 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe90153670 0x7fbe900d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe90153670 0x7fbe900d1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47940/0 (socket says 172.21.15.70:47940) 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 -- 172.21.15.70:0/2228490442 learned_addr learned my addr 172.21.15.70:0/2228490442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 -- 172.21.15.70:0/2228490442 >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 msgr2=0x7fbe900cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe953df640 1 --2- 172.21.15.70:0/2228490442 >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 0x7fbe900cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 --2- 172.21.15.70:0/2228490442 >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 0x7fbe900cf160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 -- 172.21.15.70:0/2228490442 >> v1:172.21.15.96:6789/0 conn(0x7fbe900d21e0 legacy=0x7fbe900d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe94bde640 1 -- 172.21.15.70:0/2228490442 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9006c240 con 0x7fbe90153670 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.663+0000 7fbe953df640 1 --2- 172.21.15.70:0/2228490442 >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 0x7fbe900cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.664+0000 7fbe94bde640 1 --2- 172.21.15.70:0/2228490442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe900d1ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe900ceff0 tx=0x7fbe8402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3df489100b096bb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.664+0000 7fbe8f7fe640 1 -- 172.21.15.70:0/2228490442 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8402e070 con 0x7fbe90153670 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.664+0000 7fbe8f7fe640 1 -- 172.21.15.70:0/2228490442 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe84037070 con 0x7fbe90153670 2024-04-15T15:15:03.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.664+0000 7fbe8f7fe640 1 -- 172.21.15.70:0/2228490442 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84031a00 con 0x7fbe90153670 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 -- 172.21.15.70:0/2228490442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 msgr2=0x7fbe900d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 --2- 172.21.15.70:0/2228490442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe900d1ca0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe900ceff0 tx=0x7fbe8402b880 comp rx=0 tx=0).stop 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 -- 172.21.15.70:0/2228490442 shutdown_connections 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 --2- 172.21.15.70:0/2228490442 >> v2:172.21.15.96:3300/0 conn(0x7fbe900d0dc0 0x7fbe900cf160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 --2- 172.21.15.70:0/2228490442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe900d1ca0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 -- 172.21.15.70:0/2228490442 >> 172.21.15.70:0/2228490442 conn(0x7fbe9009a9d0 msgr2=0x7fbe900d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 -- 172.21.15.70:0/2228490442 shutdown_connections 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.665+0000 7fbe96e42640 1 -- 172.21.15.70:0/2228490442 wait complete. 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.666+0000 7fbe96e42640 1 Processor -- start 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.666+0000 7fbe96e42640 1 -- start start 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe900d0dc0 0x7fbe9015f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe900d21e0 0x7fbe9006f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9015db40 con 0x7fbe90153670 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9015d9c0 con 0x7fbe900d21e0 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe96e42640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbe900cf6a0 con 0x7fbe900d0dc0 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47948/0 (socket says 172.21.15.70:47948) 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 -- 172.21.15.70:0/4037073381 learned_addr learned my addr 172.21.15.70:0/4037073381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe94bde640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe900d0dc0 0x7fbe9015f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe8ffff640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe900d21e0 0x7fbe9006f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 -- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe900d0dc0 msgr2=0x7fbe9015f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe900d0dc0 0x7fbe9015f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 -- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe900d21e0 msgr2=0x7fbe9006f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe900d21e0 0x7fbe9006f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 -- 172.21.15.70:0/4037073381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe9006fe70 con 0x7fbe90153670 2024-04-15T15:15:03.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.667+0000 7fbe953df640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800cf20 tx=0x7fbe8800a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8801a070 con 0x7fbe90153670 2024-04-15T15:15:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe8800d840 con 0x7fbe90153670 2024-04-15T15:15:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe90070150 con 0x7fbe90153670 2024-04-15T15:15:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8800dc30 con 0x7fbe90153670 2024-04-15T15:15:03.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe90070710 con 0x7fbe90153670 2024-04-15T15:15:03.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.668+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe900690f0 con 0x7fbe90153670 2024-04-15T15:15:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.669+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe8801d070 con 0x7fbe90153670 2024-04-15T15:15:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.669+0000 7fbe8d7fa640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 0x7fbe7406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.669+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fbe8808bff0 con 0x7fbe90153670 2024-04-15T15:15:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.669+0000 7fbe94bde640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 0x7fbe7406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.670+0000 7fbe94bde640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 0x7fbe7406edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.674+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe8805a120 con 0x7fbe90153670 2024-04-15T15:15:03.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.850+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe900cc9d0 con 0x7fbe90153670 2024-04-15T15:15:03.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.851+0000 7fbe8d7fa640 1 -- 172.21.15.70:0/4037073381 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12824 (secure 0 0 0) 0x7fbe8805e420 con 0x7fbe90153670 2024-04-15T15:15:03.852 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:03.853 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:00.252437+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 msgr2=0x7fbe7406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 0x7fbe7406edb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbe84036000 tx=0x7fbe8403d040 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 msgr2=0x7fbe90155e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800cf20 tx=0x7fbe8800a7d0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 shutdown_connections 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe7406c8f0 0x7fbe7406edb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe90153670 0x7fbe90155e30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe900d21e0 0x7fbe9006f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 --2- 172.21.15.70:0/4037073381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe900d0dc0 0x7fbe9015f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 >> 172.21.15.70:0/4037073381 conn(0x7fbe9009a9d0 msgr2=0x7fbe900d06b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 shutdown_connections 2024-04-15T15:15:03.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.853+0000 7fbe96e42640 1 -- 172.21.15.70:0/4037073381 wait complete. 2024-04-15T15:15:03.866 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:15:03.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.973+0000 7f11257ba640 1 Processor -- start 2024-04-15T15:15:03.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 -- start start 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1120071d10 0x7f11200d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11200d0bd0 0x7f112006e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f112015db40 con 0x7f1120072d80 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1120072170 con 0x7f1120071d10 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11257ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f112015cfc0 con 0x7f11200d0bd0 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.974+0000 7f11167fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1120071d10 0x7f11200d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1120071d10 0x7f11200d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47964/0 (socket says 172.21.15.70:47964) 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 -- 172.21.15.70:0/881127950 learned_addr learned my addr 172.21.15.70:0/881127950 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f111f7fe640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11200d0bd0 0x7f112006e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 -- 172.21.15.70:0/881127950 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11200d0bd0 msgr2=0x7f112006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11200d0bd0 0x7f112006e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 -- 172.21.15.70:0/881127950 >> v1:172.21.15.96:6789/0 conn(0x7f1120072d80 legacy=0x7f11200af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.975+0000 7f11167fc640 1 -- 172.21.15.70:0/881127950 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f112015d2a0 con 0x7f1120071d10 2024-04-15T15:15:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11167fc640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1120071d10 0x7f11200d0690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1108009870 tx=0x7f110802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10137d839edce4a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f111e7fc640 1 -- 172.21.15.70:0/881127950 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1108035020 con 0x7f1120071d10 2024-04-15T15:15:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f111e7fc640 1 -- 172.21.15.70:0/881127950 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1108031640 con 0x7f1120071d10 2024-04-15T15:15:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f111e7fc640 1 -- 172.21.15.70:0/881127950 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110802d040 con 0x7f1120071d10 2024-04-15T15:15:03.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 -- 172.21.15.70:0/881127950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1120071d10 msgr2=0x7f11200d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1120071d10 0x7f11200d0690 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1108009870 tx=0x7f110802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 -- 172.21.15.70:0/881127950 shutdown_connections 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11200d0bd0 0x7f112006e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 --2- 172.21.15.70:0/881127950 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1120071d10 0x7f11200d0690 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 -- 172.21.15.70:0/881127950 >> 172.21.15.70:0/881127950 conn(0x7f112006be00 msgr2=0x7f112006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:03.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 -- 172.21.15.70:0/881127950 shutdown_connections 2024-04-15T15:15:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.976+0000 7f11257ba640 1 -- 172.21.15.70:0/881127950 wait complete. 2024-04-15T15:15:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.977+0000 7f11257ba640 1 Processor -- start 2024-04-15T15:15:03.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 -- start start 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1120071d10 0x7f1120157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1120072d80 0x7f1120157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11200afaf0 con 0x7f11200d0bd0 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f112015d860 con 0x7f1120071d10 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.978+0000 7f11257ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f112015cb90 con 0x7f1120072d80 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47968/0 (socket says 172.21.15.70:47968) 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 -- 172.21.15.70:0/2621474730 learned_addr learned my addr 172.21.15.70:0/2621474730 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f11167fc640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1120072d80 0x7f1120157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111effd640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1120071d10 0x7f1120157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 -- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1120072d80 msgr2=0x7f1120157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1120072d80 0x7f1120157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 -- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1120071d10 msgr2=0x7f1120157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1120071d10 0x7f1120157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 -- 172.21.15.70:0/2621474730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11201f2f40 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f111f7fe640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f111800efc0 tx=0x7f111800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118018020 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1118014480 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11200cce20 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1118014870 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.979+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11200cd3e0 con 0x7f11200d0bd0 2024-04-15T15:15:03.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.980+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11200689e0 con 0x7f11200d0bd0 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.981+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1118005d10 con 0x7f11200d0bd0 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.981+0000 7f1117fff640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 0x7f110006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.981+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f111808c240 con 0x7f11200d0bd0 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.981+0000 7f111effd640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 0x7f110006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.981+0000 7f111effd640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 0x7f110006edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11200ad640 tx=0x7f1110009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:03.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:03.985+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1118022070 con 0x7f11200d0bd0 2024-04-15T15:15:04.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.161+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f11200ccc00 con 0x7f11200d0bd0 2024-04-15T15:15:04.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.162+0000 7f1117fff640 1 -- 172.21.15.70:0/2621474730 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f1118010040 con 0x7f11200d0bd0 2024-04-15T15:15:04.164 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 msgr2=0x7f110006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 0x7f110006edb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11200ad640 tx=0x7f1110009040 comp rx=0 tx=0).stop 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 msgr2=0x7f11201f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f111800efc0 tx=0x7f111800c490 comp rx=0 tx=0).stop 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 shutdown_connections 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f110006c8f0 0x7f110006edb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11200d0bd0 0x7f11201f21b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1120072d80 0x7f1120157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 --2- 172.21.15.70:0/2621474730 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1120071d10 0x7f1120157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 >> 172.21.15.70:0/2621474730 conn(0x7f112006be00 msgr2=0x7f11200b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 shutdown_connections 2024-04-15T15:15:04.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.164+0000 7f11257ba640 1 -- 172.21.15.70:0/2621474730 wait complete. 2024-04-15T15:15:04.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.278+0000 7f2851a35640 1 Processor -- start 2024-04-15T15:15:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.278+0000 7f2851a35640 1 -- start start 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f2851a35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f284c070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f2851a35640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284c06fd90 con 0x7f284c0cf600 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f2851a35640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f284c0ae440 con 0x7f284c0b04f0 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f2851a35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f284c0ae720 con 0x7f284c15ed00 2024-04-15T15:15:04.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f284a7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f284c0cf600 0x7f284c06ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33366/0 (socket says 172.21.15.70:33366) 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.279+0000 7f284a7fc640 1 -- 172.21.15.70:0/3326062898 learned_addr learned my addr 172.21.15.70:0/3326062898 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f284c070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955310661 0 0) 0x7f284c06fd90 con 0x7f284c0cf600 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2834003520 con 0x7f284c0cf600 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2601840600 0 0) 0x7f284c0ae440 con 0x7f284c0b04f0 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f284c06fd90 con 0x7f284c0b04f0 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3174674727 0 0) 0x7f2834003520 con 0x7f284c0cf600 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 msgr2=0x7f284c070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f284c070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 >> v1:172.21.15.96:6789/0 conn(0x7f284c0b04f0 legacy=0x7f284c06f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c0aea00 con 0x7f284c0cf600 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f2840001670 con 0x7f284c0cf600 2024-04-15T15:15:04.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 >> v1:172.21.15.70:6789/0 conn(0x7f284c0cf600 legacy=0x7f284c06ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2834003b70 0x7f2834003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834004800 0x7f2834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f28340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f284c0ae720 con 0x7f2834003b70 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f283400ba70 con 0x7f2834004800 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f283400bc70 con 0x7f284c15ed00 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2834003b70 0x7f2834003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284a7fc640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834004800 0x7f2834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284b7fe640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f28340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 -- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 msgr2=0x7f28340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15ed00 0x7f28340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 -- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834004800 msgr2=0x7f2834004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834004800 0x7f2834004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.280+0000 7f284affd640 1 -- 172.21.15.70:0/3326062898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2834003520 con 0x7f2834003b70 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.281+0000 7f284affd640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2834003b70 0x7f2834003f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f283c009fc0 tx=0x7f2834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcccb114d1cbbdec server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.281+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c039070 con 0x7f2834003b70 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.281+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f283c02cc20 con 0x7f2834003b70 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.281+0000 7f2849ffb640 1 -- 172.21.15.70:0/3326062898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f283c009c10 con 0x7f2834003b70 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 -- 172.21.15.70:0/3326062898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2834003b70 msgr2=0x7f2834003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 --2- 172.21.15.70:0/3326062898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2834003b70 0x7f2834003f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f283c009fc0 tx=0x7f2834005830 comp rx=0 tx=0).stop 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f284b7fe640 1 -- 172.21.15.70:0/3326062898 reap_dead start 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 -- 172.21.15.70:0/3326062898 shutdown_connections 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 -- 172.21.15.70:0/3326062898 >> 172.21.15.70:0/3326062898 conn(0x7f284c09a9d0 msgr2=0x7f284c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 -- 172.21.15.70:0/3326062898 shutdown_connections 2024-04-15T15:15:04.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.282+0000 7f2851a35640 1 -- 172.21.15.70:0/3326062898 wait complete. 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.283+0000 7f2851a35640 1 Processor -- start 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.283+0000 7f2851a35640 1 -- start start 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834003b70 0x7f284c153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15a8e0 0x7f284c15ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f284c0701a0 con 0x7f284c0b4950 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f284c0aefc0 con 0x7f2834003b70 2024-04-15T15:15:04.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f2851a35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f284c0cfa60 con 0x7f284c15a8e0 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47988/0 (socket says 172.21.15.70:47988) 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 -- 172.21.15.70:0/2995736870 learned_addr learned my addr 172.21.15.70:0/2995736870 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284affd640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834003b70 0x7f284c153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284a7fc640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15a8e0 0x7f284c15ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 -- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15a8e0 msgr2=0x7f284c15ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15a8e0 0x7f284c15ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 -- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834003b70 msgr2=0x7f284c153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834003b70 0x7f284c153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 -- 172.21.15.70:0/2995736870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f284c0b5310 con 0x7f284c0b4950 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.284+0000 7f284b7fe640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f284c0c5070 tx=0x7f284400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.285+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440050d0 con 0x7f284c0b4950 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.285+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2844018070 con 0x7f284c0b4950 2024-04-15T15:15:04.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.285+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28440035f0 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.285+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f284c0b6bd0 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.285+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f284c0b7190 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.286+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284c0690f0 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.286+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2844003790 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.287+0000 7f2833fff640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 0x7f283806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.287+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f284408b090 con 0x7f284c0b4950 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.287+0000 7f284affd640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 0x7f283806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:04.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.287+0000 7f284affd640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 0x7f283806ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f283c036000 tx=0x7f283c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:04.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.291+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2844022020 con 0x7f284c0b4950 2024-04-15T15:15:04.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.502+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f284c0b0160 con 0x7f284c0b4950 2024-04-15T15:15:04.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.503+0000 7f2833fff640 1 -- 172.21.15.70:0/2995736870 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7f2844059250 con 0x7f284c0b4950 2024-04-15T15:15:04.505 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:04.505 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-15T15:15:04.505 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 2 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 msgr2=0x7f283806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 0x7f283806ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f283c036000 tx=0x7f283c02e000 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 msgr2=0x7f284c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f284c0c5070 tx=0x7f284400fc00 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 shutdown_connections 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f283806c920 0x7f283806ede0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f284c0b4950 0x7f284c0b4dd0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f284c15a8e0 0x7f284c15ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 --2- 172.21.15.70:0/2995736870 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2834003b70 0x7f284c153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 >> 172.21.15.70:0/2995736870 conn(0x7f284c09a9d0 msgr2=0x7f284c0b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 shutdown_connections 2024-04-15T15:15:04.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:04.505+0000 7f2851a35640 1 -- 172.21.15.70:0/2995736870 wait complete. 2024-04-15T15:15:06.518 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:06.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.631+0000 7faba3ba8640 1 Processor -- start 2024-04-15T15:15:06.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.631+0000 7faba3ba8640 1 -- start start 2024-04-15T15:15:06.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba3ba8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c09a9d0 0x7fab9c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.634 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba3ba8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fab9c0701f0 0x7fab9c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba3ba8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab9c070720 con 0x7fab9c09adb0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba3ba8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fab9c09ba90 con 0x7fab9c0701f0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba3ba8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c160070 con 0x7fab9c09a9d0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba1143640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fab9c0701f0 0x7fab9c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba2145640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fab9c09adb0 0x7fab9c0d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33382/0 (socket says 172.21.15.70:33382) 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.632+0000 7faba2145640 1 -- 172.21.15.70:0/4053793174 learned_addr learned my addr 172.21.15.70:0/4053793174 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1944640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c09a9d0 0x7fab9c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1143640 1 -- 172.21.15.70:0/4053793174 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c09a9d0 msgr2=0x7fab9c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1143640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c09a9d0 0x7fab9c160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1143640 1 -- 172.21.15.70:0/4053793174 >> v1:172.21.15.70:6789/0 conn(0x7fab9c09adb0 legacy=0x7fab9c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1143640 1 -- 172.21.15.70:0/4053793174 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c070a90 con 0x7fab9c0701f0 2024-04-15T15:15:06.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba0942640 1 -- 172.21.15.70:0/4053793174 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416824081 0 0) 0x7fab9c070720 con 0x7fab9c09adb0 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba1143640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c0701f0 0x7fab9c15fb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0097d0 tx=0x7fab8c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=992547ded8904c18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.633+0000 7faba0942640 1 -- 172.21.15.70:0/4053793174 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c035020 con 0x7fab9c0701f0 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba0942640 1 -- 172.21.15.70:0/4053793174 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab8c031800 con 0x7fab9c0701f0 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba0942640 1 -- 172.21.15.70:0/4053793174 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab8c031bd0 con 0x7fab9c0701f0 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 -- 172.21.15.70:0/4053793174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c0701f0 msgr2=0x7fab9c15fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c0701f0 0x7fab9c15fb30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0097d0 tx=0x7fab8c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 -- 172.21.15.70:0/4053793174 shutdown_connections 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c0701f0 0x7fab9c15fb30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 --2- 172.21.15.70:0/4053793174 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c09a9d0 0x7fab9c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 -- 172.21.15.70:0/4053793174 >> 172.21.15.70:0/4053793174 conn(0x7fab9c0d2460 msgr2=0x7fab9c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:06.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 -- 172.21.15.70:0/4053793174 shutdown_connections 2024-04-15T15:15:06.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.634+0000 7faba3ba8640 1 -- 172.21.15.70:0/4053793174 wait complete. 2024-04-15T15:15:06.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.635+0000 7faba3ba8640 1 Processor -- start 2024-04-15T15:15:06.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.635+0000 7faba3ba8640 1 -- start start 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 0x7fab9c0c6550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab9c09a9d0 0x7fab9c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c071050 con 0x7fab9c09adb0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c0d08c0 con 0x7fab9c09a9d0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba3ba8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c0b1c60 con 0x7fab9c0701f0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba2145640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba2145640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47994/0 (socket says 172.21.15.70:47994) 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba2145640 1 -- 172.21.15.70:0/135418477 learned_addr learned my addr 172.21.15.70:0/135418477 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.636+0000 7faba1944640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 0x7fab9c0c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 -- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 msgr2=0x7fab9c0c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba1143640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab9c09a9d0 0x7fab9c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 0x7fab9c0c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 -- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab9c09a9d0 msgr2=0x7fab9c06dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab9c09a9d0 0x7fab9c06dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 -- 172.21.15.70:0/135418477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c14d410 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba1944640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 0x7fab9c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba2145640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c450 tx=0x7fab9400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94018070 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab94007500 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c14d6f0 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab94004a20 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.637+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c14d0b0 con 0x7fab9c09adb0 2024-04-15T15:15:06.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.638+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c0689e0 con 0x7fab9c09adb0 2024-04-15T15:15:06.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.639+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fab9401b070 con 0x7fab9c09adb0 2024-04-15T15:15:06.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.639+0000 7fab927fc640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 0x7fab8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.639+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fab9408aa40 con 0x7fab9c09adb0 2024-04-15T15:15:06.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.639+0000 7faba1944640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 0x7fab8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.639+0000 7faba1944640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 0x7fab8006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab98007f50 tx=0x7fab980074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.644+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fab94059250 con 0x7fab9c09adb0 2024-04-15T15:15:06.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.854+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab9c1601f0 con 0x7fab9c09adb0 2024-04-15T15:15:06.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.855+0000 7fab927fc640 1 -- 172.21.15.70:0/135418477 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3861 (secure 0 0 0) 0x7fab9408ae20 con 0x7fab9c09adb0 2024-04-15T15:15:06.857 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:06.857 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4293,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-15T15:15:06.857 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 3 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 msgr2=0x7fab8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 0x7fab8006edb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fab98007f50 tx=0x7fab980074a0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 msgr2=0x7fab9c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fab9400c450 tx=0x7fab9400c910 comp rx=0 tx=0).stop 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 shutdown_connections 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fab8006c8f0 0x7fab8006edb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab9c09adb0 0x7fab9c06e320 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fab9c09a9d0 0x7fab9c06dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 --2- 172.21.15.70:0/135418477 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fab9c0701f0 0x7fab9c0c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 >> 172.21.15.70:0/135418477 conn(0x7fab9c0d2460 msgr2=0x7fab9c15d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:06.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 shutdown_connections 2024-04-15T15:15:06.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.857+0000 7faba3ba8640 1 -- 172.21.15.70:0/135418477 wait complete. 2024-04-15T15:15:06.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.970+0000 7f469ad9f640 1 Processor -- start 2024-04-15T15:15:06.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.971+0000 7f469ad9f640 1 -- start start 2024-04-15T15:15:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.971+0000 7f469ad9f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4694145f60 0x7f46940adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.971+0000 7f469ad9f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46940b3a80 con 0x7f469406b9e0 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f469ad9f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f469415e650 con 0x7f469409aab0 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f469ad9f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f469415ef80 con 0x7f4694145f60 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f4699d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4694145f60 0x7f46940adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f469a59e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f469406b9e0 0x7f469406bdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33398/0 (socket says 172.21.15.70:33398) 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f469a59e640 1 -- 172.21.15.70:0/4125528837 learned_addr learned my addr 172.21.15.70:0/4125528837 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f4699d9d640 1 -- 172.21.15.70:0/4125528837 >> v1:172.21.15.96:6789/0 conn(0x7f469409aab0 legacy=0x7f46940af8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f4699d9d640 1 -- 172.21.15.70:0/4125528837 >> v1:172.21.15.70:6789/0 conn(0x7f469406b9e0 legacy=0x7f469406bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f4699d9d640 1 -- 172.21.15.70:0/4125528837 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469415e930 con 0x7f4694145f60 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.972+0000 7f4698d9b640 1 -- 172.21.15.70:0/4125528837 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404083026 0 0) 0x7f46940b3a80 con 0x7f469406b9e0 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f4699d9d640 1 --2- 172.21.15.70:0/4125528837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4694145f60 0x7f46940adfd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46900097a0 tx=0x7f469002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=817ef1c2e46aba18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f4698d9b640 1 -- 172.21.15.70:0/4125528837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4690034070 con 0x7f4694145f60 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f4698d9b640 1 -- 172.21.15.70:0/4125528837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4690030610 con 0x7f4694145f60 2024-04-15T15:15:06.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f4698d9b640 1 -- 172.21.15.70:0/4125528837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46900309e0 con 0x7f4694145f60 2024-04-15T15:15:06.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 -- 172.21.15.70:0/4125528837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4694145f60 msgr2=0x7f46940adfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 --2- 172.21.15.70:0/4125528837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4694145f60 0x7f46940adfd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46900097a0 tx=0x7f469002b960 comp rx=0 tx=0).stop 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 -- 172.21.15.70:0/4125528837 shutdown_connections 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 --2- 172.21.15.70:0/4125528837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4694145f60 0x7f46940adfd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 -- 172.21.15.70:0/4125528837 >> 172.21.15.70:0/4125528837 conn(0x7f4694073060 msgr2=0x7f46940b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 -- 172.21.15.70:0/4125528837 shutdown_connections 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.973+0000 7f469ad9f640 1 -- 172.21.15.70:0/4125528837 wait complete. 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 Processor -- start 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 -- start start 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469406b9e0 0x7f469413c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4694145f60 0x7f46941e5540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46940affd0 con 0x7f469409aab0 2024-04-15T15:15:06.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46940b3720 con 0x7f469406b9e0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469ad9f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f469415f410 con 0x7f4694145f60 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469959c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469959c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48014/0 (socket says 172.21.15.70:48014) 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.975+0000 7f469959c640 1 -- 172.21.15.70:0/3805783166 learned_addr learned my addr 172.21.15.70:0/3805783166 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469a59e640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4694145f60 0x7f46941e5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f4699d9d640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469406b9e0 0x7f469413c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 -- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4694145f60 msgr2=0x7f46941e5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4694145f60 0x7f46941e5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 -- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469406b9e0 msgr2=0x7f469413c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469406b9e0 0x7f469413c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 -- 172.21.15.70:0/3805783166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f469413e9e0 con 0x7f469409aab0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469959c640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f468400c400 tx=0x7f468400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684017070 con 0x7f469409aab0 2024-04-15T15:15:06.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46840052d0 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f469413ecc0 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4684003830 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.976+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f469413f280 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.977+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4694145100 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.977+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f468401a070 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.978+0000 7f468a7fc640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 0x7f467806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.978+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f468408ab90 con 0x7f469409aab0 2024-04-15T15:15:06.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.978+0000 7f4699d9d640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 0x7f467806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:06.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.978+0000 7f4699d9d640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 0x7f467806ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f4690008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:06.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:06.982+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4694145100 con 0x7f469409aab0 2024-04-15T15:15:07.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.160+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46941e9340 con 0x7f469409aab0 2024-04-15T15:15:07.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.161+0000 7f468a7fc640 1 -- 172.21.15.70:0/3805783166 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12824 (secure 0 0 0) 0x7f4684059310 con 0x7f469409aab0 2024-04-15T15:15:07.163 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:07.163 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:00.252437+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:07.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 msgr2=0x7f467806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 0x7f467806ede0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4690036000 tx=0x7f4690008030 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 msgr2=0x7f469413dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f468400c400 tx=0x7f468400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 shutdown_connections 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4694145f60 0x7f46941e5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f467806c920 0x7f467806ede0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f469409aab0 0x7f469413dfa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 --2- 172.21.15.70:0/3805783166 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f469406b9e0 0x7f469413c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 >> 172.21.15.70:0/3805783166 conn(0x7f4694073060 msgr2=0x7f469413b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 shutdown_connections 2024-04-15T15:15:07.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.163+0000 7f469ad9f640 1 -- 172.21.15.70:0/3805783166 wait complete. 2024-04-15T15:15:07.175 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:15:07.176 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:07.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.280+0000 7fb8bf1c3640 1 Processor -- start 2024-04-15T15:15:07.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.280+0000 7fb8bf1c3640 1 -- start start 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bf1c3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bf1c3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b8067e30 con 0x7fb8b815e830 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bf1c3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b80b3c10 con 0x7fb8b815ec10 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bf1c3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b80b3ef0 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bd760640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bcf5f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb8b815e830 0x7fb8b8067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33408/0 (socket says 172.21.15.70:33408) 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bd760640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48020/0 (socket says 172.21.15.70:48020) 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bcf5f640 1 -- 172.21.15.70:0/189521834 learned_addr learned my addr 172.21.15.70:0/189521834 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bd760640 1 -- 172.21.15.70:0/189521834 >> v1:172.21.15.96:6789/0 conn(0x7fb8b815ec10 legacy=0x7fb8b80b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bd760640 1 -- 172.21.15.70:0/189521834 >> v1:172.21.15.70:6789/0 conn(0x7fb8b815e830 legacy=0x7fb8b8067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8b77fe640 1 -- 172.21.15.70:0/189521834 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497435425 0 0) 0x7fb8b8067e30 con 0x7fb8b815e830 2024-04-15T15:15:07.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.281+0000 7fb8bd760640 1 -- 172.21.15.70:0/189521834 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b80b41d0 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bd760640 1 --2- 172.21.15.70:0/189521834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009d90 tx=0x7fb8b002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8239c25033236c3f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8b77fe640 1 -- 172.21.15.70:0/189521834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0039070 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8b77fe640 1 -- 172.21.15.70:0/189521834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b002cbd0 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8b77fe640 1 -- 172.21.15.70:0/189521834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0034440 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/189521834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b5ad0 msgr2=0x7fb8b80b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/189521834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009d90 tx=0x7fb8b002be20 comp rx=0 tx=0).stop 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/189521834 shutdown_connections 2024-04-15T15:15:07.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/189521834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b80b5ad0 0x7fb8b80b4cb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/189521834 >> 172.21.15.70:0/189521834 conn(0x7fb8b8073360 msgr2=0x7fb8b806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/189521834 shutdown_connections 2024-04-15T15:15:07.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.282+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/189521834 wait complete. 2024-04-15T15:15:07.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.283+0000 7fb8bf1c3640 1 Processor -- start 2024-04-15T15:15:07.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 -- start start 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b80b5ad0 0x7fb8b8160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b815e830 0x7fb8b81587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b815ff60 con 0x7fb8b81685f0 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b80b4790 con 0x7fb8b815e830 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bf1c3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b81600e0 con 0x7fb8b80b5ad0 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.284+0000 7fb8bd760640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48030/0 (socket says 172.21.15.70:48030) 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 -- 172.21.15.70:0/1475231687 learned_addr learned my addr 172.21.15.70:0/1475231687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 -- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b80b5ad0 msgr2=0x7fb8b8160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b80b5ad0 0x7fb8b8160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 -- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b815e830 msgr2=0x7fb8b81587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b815e830 0x7fb8b81587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 -- 172.21.15.70:0/1475231687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b8158f10 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bd760640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009d90 tx=0x7fb8b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0044020 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8b003f470 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b81591f0 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0039040 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.285+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b81597b0 con 0x7fb8b81685f0 2024-04-15T15:15:07.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.286+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b80689e0 con 0x7fb8b81685f0 2024-04-15T15:15:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.287+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8b0047020 con 0x7fb8b81685f0 2024-04-15T15:15:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.287+0000 7fb8b57fa640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 0x7fb89c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.287+0000 7fb8bcf5f640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 0x7fb89c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.287+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb8b00ae420 con 0x7fb8b81685f0 2024-04-15T15:15:07.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.288+0000 7fb8bcf5f640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 0x7fb89c06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac007fd0 tx=0x7fb8ac007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.292+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8b004e030 con 0x7fb8b81685f0 2024-04-15T15:15:07.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.465+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb8b80af5c0 con 0x7fb8b81685f0 2024-04-15T15:15:07.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.465+0000 7fb8b57fa640 1 -- 172.21.15.70:0/1475231687 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7fb8b007cba0 con 0x7fb8b81685f0 2024-04-15T15:15:07.467 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 msgr2=0x7fb89c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 0x7fb89c06edb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac007fd0 tx=0x7fb8ac007920 comp rx=0 tx=0).stop 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 msgr2=0x7fb8b8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb8b0009d90 tx=0x7fb8b0004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 shutdown_connections 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89c06c8f0 0x7fb89c06edb0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b81685f0 0x7fb8b8159fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b815e830 0x7fb8b81587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 --2- 172.21.15.70:0/1475231687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b80b5ad0 0x7fb8b8160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 >> 172.21.15.70:0/1475231687 conn(0x7fb8b8073360 msgr2=0x7fb8b80afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 shutdown_connections 2024-04-15T15:15:07.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.467+0000 7fb8bf1c3640 1 -- 172.21.15.70:0/1475231687 wait complete. 2024-04-15T15:15:07.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:15:07.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 Processor -- start 2024-04-15T15:15:07.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 -- start start 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f68200cf600 0x7f68200cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68200ae440 con 0x7f68200af4e0 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68200ae720 con 0x7f682015ed00 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.581+0000 7f6825c16640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6820071030 con 0x7f68200cf600 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f68200af4e0 0x7f68200af910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33422/0 (socket says 172.21.15.70:33422) 2024-04-15T15:15:07.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681ffff640 1 -- 172.21.15.70:0/2925276668 learned_addr learned my addr 172.21.15.70:0/2925276668 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2367221050 0 0) 0x7f68200ae440 con 0x7f68200af4e0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f680c003520 con 0x7f68200af4e0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f6816ffd640 1 --2- 172.21.15.70:0/2925276668 >> v2:172.21.15.96:3300/0 conn(0x7f68200cf600 0x7f68200cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 115799318 0 0) 0x7f680c003520 con 0x7f68200af4e0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 >> v2:172.21.15.96:3300/0 conn(0x7f68200cf600 msgr2=0x7f68200cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 --2- 172.21.15.70:0/2925276668 >> v2:172.21.15.96:3300/0 conn(0x7f68200cf600 0x7f68200cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 >> v1:172.21.15.96:6789/0 conn(0x7f682015ed00 legacy=0x7f6820070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68200aea00 con 0x7f68200af4e0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f68180019f0 con 0x7f68200af4e0 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 >> v1:172.21.15.70:6789/0 conn(0x7f68200af4e0 legacy=0x7f68200af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f680c0039b0 0x7f680c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c004800 0x7f680c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 0x7f680c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f680c00c030 con 0x7f680c0051c0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f680c00c1e0 con 0x7f680c004800 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f680c00c3e0 con 0x7f680c0039b0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681ffff640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 0x7f680c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f681f7fe640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f680c0039b0 0x7f680c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.582+0000 7f6816ffd640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c004800 0x7f680c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 -- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f680c0039b0 msgr2=0x7f680c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f680c0039b0 0x7f680c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 -- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c004800 msgr2=0x7f680c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c004800 0x7f680c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 -- 172.21.15.70:0/2925276668 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f680c003520 con 0x7f680c0051c0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 -- 172.21.15.70:0/2925276668 reap_dead start 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681ffff640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 0x7f680c00b9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f681800a850 tx=0x7f680c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63fc628b0d634bfc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f681800a4a0 con 0x7f680c0051c0 2024-04-15T15:15:07.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818034650 con 0x7f680c0051c0 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.583+0000 7f681effd640 1 -- 172.21.15.70:0/2925276668 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818034a40 con 0x7f680c0051c0 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 -- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 msgr2=0x7f680c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 0x7f680c00b9c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f681800a850 tx=0x7f680c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 -- 172.21.15.70:0/2925276668 shutdown_connections 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 --2- 172.21.15.70:0/2925276668 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f680c0051c0 0x7f680c00b9c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 -- 172.21.15.70:0/2925276668 >> 172.21.15.70:0/2925276668 conn(0x7f682009a9d0 msgr2=0x7f68200d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 -- 172.21.15.70:0/2925276668 shutdown_connections 2024-04-15T15:15:07.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.584+0000 7f6825c16640 1 -- 172.21.15.70:0/2925276668 wait complete. 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.585+0000 7f6825c16640 1 Processor -- start 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.585+0000 7f6825c16640 1 -- start start 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 0x7f68201f7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 0x7f68200c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68200701a0 con 0x7f68200c6470 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68200aefc0 con 0x7f680c0051c0 2024-04-15T15:15:07.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6825c16640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f68200cfa90 con 0x7f68201f4150 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48038/0 (socket says 172.21.15.70:48038) 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 -- 172.21.15.70:0/2076482003 learned_addr learned my addr 172.21.15.70:0/2076482003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6816ffd640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 0x7f68200c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681f7fe640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 0x7f68201f7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 -- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 msgr2=0x7f68200c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 0x7f68200c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 -- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 msgr2=0x7f68201f7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 0x7f68201f7720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681ffff640 1 -- 172.21.15.70:0/2076482003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68201f85a0 con 0x7f68200c6470 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f6816ffd640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 0x7f68200c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.586+0000 7f681f7fe640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 0x7f68201f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:07.594 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f681ffff640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f681800a850 tx=0x7f6818001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6818039040 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6818043070 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68201f7d00 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68180017b0 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.587+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68201f82c0 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.588+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68200690f0 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.588+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6818001950 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.588+0000 7f681cff9640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 0x7f680006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.589+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f68180ad540 con 0x7f68200c6470 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.589+0000 7f681f7fe640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 0x7f680006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.589+0000 7f681f7fe640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 0x7f680006ede0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68200b53a0 tx=0x7f6810009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.593+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f681807bcc0 con 0x7f68200c6470 2024-04-15T15:15:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.769+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f68200752d0 con 0x7f68200c6470 2024-04-15T15:15:07.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.770+0000 7f681cff9640 1 -- 172.21.15.70:0/2076482003 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f681807b5c0 con 0x7f68200c6470 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:15:07.772 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-15T15:15:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 msgr2=0x7f680006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 0x7f680006ede0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68200b53a0 tx=0x7f6810009040 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 msgr2=0x7f68200afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f681800a850 tx=0x7f6818001dc0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 shutdown_connections 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f680006c920 0x7f680006ede0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68200c6470 0x7f68200afde0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68201f4150 0x7f68200c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 --2- 172.21.15.70:0/2076482003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f680c0051c0 0x7f68201f7720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.771+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 >> 172.21.15.70:0/2076482003 conn(0x7f682009a9d0 msgr2=0x7f68200ccd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.772+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 shutdown_connections 2024-04-15T15:15:07.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.772+0000 7f6825c16640 1 -- 172.21.15.70:0/2076482003 wait complete. 2024-04-15T15:15:07.784 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:15:07.784 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:15:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.889+0000 7fa4d0111640 1 Processor -- start 2024-04-15T15:15:07.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.889+0000 7fa4d0111640 1 -- start start 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c81623f0 0x7fa4c80d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 0x7fa4c81627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa4c8162d10 0x7fa4c806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c80d0680 con 0x7fa4c81623f0 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa4c80d08b0 con 0x7fa4c8162d10 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4d0111640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c80d0b90 con 0x7fa4c809b3e0 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c81623f0 0x7fa4c80d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4c81623f0 0x7fa4c80d1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48054/0 (socket says 172.21.15.70:48054) 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 -- 172.21.15.70:0/3218056280 learned_addr learned my addr 172.21.15.70:0/3218056280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:07.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4ce6ae640 1 --2- 172.21.15.70:0/3218056280 >> v2:172.21.15.96:3300/0 conn(0x7fa4c8162d10 0x7fa4c806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 0x7fa4c81627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 -- 172.21.15.70:0/3218056280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 msgr2=0x7fa4c81627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 0x7fa4c81627d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 -- 172.21.15.70:0/3218056280 >> v2:172.21.15.96:3300/0 conn(0x7fa4c8162d10 msgr2=0x7fa4c806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 --2- 172.21.15.70:0/3218056280 >> v2:172.21.15.96:3300/0 conn(0x7fa4c8162d10 0x7fa4c806caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cdead640 1 -- 172.21.15.70:0/3218056280 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c80d0e70 con 0x7fa4c81623f0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.890+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 0x7fa4c81627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.891+0000 7fa4cdead640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c81623f0 0x7fa4c80d1670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80d2070 tx=0x7fa4c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d121b29cb26e27 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.891+0000 7fa4cceab640 1 -- 172.21.15.70:0/3218056280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4035020 con 0x7fa4c81623f0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.891+0000 7fa4cceab640 1 -- 172.21.15.70:0/3218056280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c4031870 con 0x7fa4c81623f0 2024-04-15T15:15:07.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.891+0000 7fa4cceab640 1 -- 172.21.15.70:0/3218056280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4031c40 con 0x7fa4c81623f0 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 -- 172.21.15.70:0/3218056280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c81623f0 msgr2=0x7fa4c80d1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c81623f0 0x7fa4c80d1670 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa4c80d2070 tx=0x7fa4c402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 -- 172.21.15.70:0/3218056280 shutdown_connections 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 --2- 172.21.15.70:0/3218056280 >> v2:172.21.15.96:3300/0 conn(0x7fa4c8162d10 0x7fa4c806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c809b3e0 0x7fa4c81627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 --2- 172.21.15.70:0/3218056280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c81623f0 0x7fa4c80d1670 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 -- 172.21.15.70:0/3218056280 >> 172.21.15.70:0/3218056280 conn(0x7fa4c80b3f70 msgr2=0x7fa4c80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 -- 172.21.15.70:0/3218056280 shutdown_connections 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.892+0000 7fa4d0111640 1 -- 172.21.15.70:0/3218056280 wait complete. 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.893+0000 7fa4d0111640 1 Processor -- start 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.893+0000 7fa4d0111640 1 -- start start 2024-04-15T15:15:07.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 0x7fa4c815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c8162d10 0x7fa4c815f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c81631a0 con 0x7fa4c809b3e0 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4c809b840 con 0x7fa4c81623f0 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4d0111640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa4c8163320 con 0x7fa4c8162d10 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48064/0 (socket says 172.21.15.70:48064) 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 -- 172.21.15.70:0/577090660 learned_addr learned my addr 172.21.15.70:0/577090660 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4ce6ae640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c8162d10 0x7fa4c815f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 0x7fa4c815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 -- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c8162d10 msgr2=0x7fa4c815f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c8162d10 0x7fa4c815f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 -- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 msgr2=0x7fa4c815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 0x7fa4c815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 -- 172.21.15.70:0/577090660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4c80b6ba0 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 0x7fa4c815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.894+0000 7fa4cdead640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40097c0 tx=0x7fa4c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.895+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4040070 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.895+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4c403a420 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.895+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4c4035040 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.895+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4c80b6e80 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.895+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4c80b7440 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.896+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4c80689e0 con 0x7fa4c809b3e0 2024-04-15T15:15:07.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.896+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa4c404b060 con 0x7fa4c809b3e0 2024-04-15T15:15:07.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.897+0000 7fa4be7fc640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 0x7fa4ac06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:07.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.897+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa4c40ad760 con 0x7fa4c809b3e0 2024-04-15T15:15:07.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.897+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 0x7fa4ac06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:07.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.897+0000 7fa4cd6ac640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 0x7fa4ac06edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:07.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:07.901+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4c407bee0 con 0x7fa4c809b3e0 2024-04-15T15:15:08.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.078+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa4c80d1c70 con 0x7fa4c809b3e0 2024-04-15T15:15:08.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.269+0000 7fa4be7fc640 1 -- 172.21.15.70:0/577090660 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fa4c407b7e0 con 0x7fa4c809b3e0 2024-04-15T15:15:08.271 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 msgr2=0x7fa4ac06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 0x7fa4ac06edb0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa4b8006fd0 tx=0x7fa4b8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 msgr2=0x7fa4c81577d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa4c40097c0 tx=0x7fa4c40046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 shutdown_connections 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa4c8162d10 0x7fa4c815f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa4c81623f0 0x7fa4c815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa4ac06c8f0 0x7fa4ac06edb0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 --2- 172.21.15.70:0/577090660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4c809b3e0 0x7fa4c81577d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 >> 172.21.15.70:0/577090660 conn(0x7fa4c80b3f70 msgr2=0x7fa4c80af4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 shutdown_connections 2024-04-15T15:15:08.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.271+0000 7fa4d0111640 1 -- 172.21.15.70:0/577090660 wait complete. 2024-04-15T15:15:08.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-15T15:15:08.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.395+0000 7f52c337c640 1 Processor -- start 2024-04-15T15:15:08.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 -- start start 2024-04-15T15:15:08.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc15ed00 0x7f52bc070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f52bc0cf600 0x7f52bc0cfa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52bc0ae440 con 0x7f52bc06ef70 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f52bc0cffa0 con 0x7f52bc0cf600 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c337c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f52bc0ae720 con 0x7f52bc15ed00 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c0917640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f52bc0cf600 0x7f52bc0cfa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.396+0000 7f52c1919640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f52bc06ef70 0x7f52bc06f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33424/0 (socket says 172.21.15.70:33424) 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c1919640 1 -- 172.21.15.70:0/3717108175 learned_addr learned my addr 172.21.15.70:0/3717108175 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c1118640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc15ed00 0x7f52bc070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c0917640 1 -- 172.21.15.70:0/3717108175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc15ed00 msgr2=0x7f52bc070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c0917640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc15ed00 0x7f52bc070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c0917640 1 -- 172.21.15.70:0/3717108175 >> v1:172.21.15.70:6789/0 conn(0x7f52bc06ef70 legacy=0x7f52bc06f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52c0917640 1 -- 172.21.15.70:0/3717108175 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc0aea00 con 0x7f52bc0cf600 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.397+0000 7f52b3fff640 1 -- 172.21.15.70:0/3717108175 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2431849597 0 0) 0x7f52bc0ae440 con 0x7f52bc06ef70 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c0917640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc0cf600 0x7f52bc0cfa60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0097a0 tx=0x7f52ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7a53802a16f8a07 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52b3fff640 1 -- 172.21.15.70:0/3717108175 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac035020 con 0x7f52bc0cf600 2024-04-15T15:15:08.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52b3fff640 1 -- 172.21.15.70:0/3717108175 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52ac031830 con 0x7f52bc0cf600 2024-04-15T15:15:08.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52b3fff640 1 -- 172.21.15.70:0/3717108175 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52ac031c00 con 0x7f52bc0cf600 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 -- 172.21.15.70:0/3717108175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc0cf600 msgr2=0x7f52bc0cfa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc0cf600 0x7f52bc0cfa60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f52ac0097a0 tx=0x7f52ac02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 -- 172.21.15.70:0/3717108175 shutdown_connections 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc0cf600 0x7f52bc0cfa60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 --2- 172.21.15.70:0/3717108175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc15ed00 0x7f52bc070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 -- 172.21.15.70:0/3717108175 >> 172.21.15.70:0/3717108175 conn(0x7f52bc09a9d0 msgr2=0x7f52bc0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 -- 172.21.15.70:0/3717108175 shutdown_connections 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.398+0000 7f52c337c640 1 -- 172.21.15.70:0/3717108175 wait complete. 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.399+0000 7f52c337c640 1 Processor -- start 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 -- start start 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc0cf600 0x7f52bc14f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52bc15ed00 0x7f52bc14f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc0701a0 con 0x7f52bc06ef70 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f52bc0aefc0 con 0x7f52bc15ed00 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c337c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f52bc070e60 con 0x7f52bc0cf600 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c1118640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c1118640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48086/0 (socket says 172.21.15.70:48086) 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.400+0000 7f52c1118640 1 -- 172.21.15.70:0/246921540 learned_addr learned my addr 172.21.15.70:0/246921540 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:08.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 -- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc0cf600 msgr2=0x7f52bc14f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc0cf600 0x7f52bc14f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 -- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52bc15ed00 msgr2=0x7f52bc14f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52bc15ed00 0x7f52bc14f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 -- 172.21.15.70:0/246921540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52bc151df0 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c1118640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f52b800d800 tx=0x7f52b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b800b6c0 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52b8019070 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52bc1f82a0 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52b800ba80 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.401+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52bc152360 con 0x7f52bc06ef70 2024-04-15T15:15:08.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.402+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52bc0690f0 con 0x7f52bc06ef70 2024-04-15T15:15:08.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.402+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f52b800bc20 con 0x7f52bc06ef70 2024-04-15T15:15:08.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.403+0000 7f52b1ffb640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 0x7f52a006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:08.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.403+0000 7f52c0917640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 0x7f52a006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:08.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.403+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f52b808bdd0 con 0x7f52bc06ef70 2024-04-15T15:15:08.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.403+0000 7f52c0917640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 0x7f52a006edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52ac036000 tx=0x7f52ac030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:08.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.408+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f52b801c360 con 0x7f52bc06ef70 2024-04-15T15:15:08.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:08.584+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f52bc0b09d0 con 0x7f52bc06ef70 2024-04-15T15:15:09.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.274+0000 7f52b1ffb640 1 -- 172.21.15.70:0/246921540 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f52b808d0d0 con 0x7f52bc06ef70 2024-04-15T15:15:09.276 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-15T15:15:09.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.278+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 msgr2=0x7f52a006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.278+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 0x7f52a006edb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f52ac036000 tx=0x7f52ac030000 comp rx=0 tx=0).stop 2024-04-15T15:15:09.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.278+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 msgr2=0x7f52bc0cdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.278+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f52b800d800 tx=0x7f52b800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 shutdown_connections 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f52bc15ed00 0x7f52bc14f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f52bc0cf600 0x7f52bc14f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f52a006c8f0 0x7f52a006edb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 --2- 172.21.15.70:0/246921540 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f52bc06ef70 0x7f52bc0cdfa0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f52b800d800 tx=0x7f52b800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.279+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 >> 172.21.15.70:0/246921540 conn(0x7f52bc09a9d0 msgr2=0x7f52bc06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.286+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 shutdown_connections 2024-04-15T15:15:09.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.287+0000 7f52c337c640 1 -- 172.21.15.70:0/246921540 wait complete. 2024-04-15T15:15:09.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:15:09.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 Processor -- start 2024-04-15T15:15:09.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 -- start start 2024-04-15T15:15:09.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0ca406e540 0x7f0ca409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca406caf0 0x7f0ca415e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca40ccc40 con 0x7f0ca406e920 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca40ccf20 con 0x7f0ca406caf0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.411+0000 7f0caab32640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca40cd200 con 0x7f0ca406e540 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca406caf0 0x7f0ca415e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca406caf0 0x7f0ca415e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48096/0 (socket says 172.21.15.70:48096) 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2373302736 learned_addr learned my addr 172.21.15.70:0/2373302736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca88ce640 1 --2- 172.21.15.70:0/2373302736 >> v2:172.21.15.96:3300/0 conn(0x7f0ca406e540 0x7f0ca409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2373302736 >> v2:172.21.15.96:3300/0 conn(0x7f0ca406e540 msgr2=0x7f0ca409b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 --2- 172.21.15.70:0/2373302736 >> v2:172.21.15.96:3300/0 conn(0x7f0ca406e540 0x7f0ca409b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2373302736 >> v1:172.21.15.96:6789/0 conn(0x7f0ca406e920 legacy=0x7f0ca40707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.412+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2373302736 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca40cd4e0 con 0x7f0ca406caf0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0ca90cf640 1 --2- 172.21.15.70:0/2373302736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406caf0 0x7f0ca415e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40739f0 tx=0x7f0c9c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1a2bc357f987cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0ca37fe640 1 -- 172.21.15.70:0/2373302736 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c02cb20 con 0x7f0ca406caf0 2024-04-15T15:15:09.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0ca37fe640 1 -- 172.21.15.70:0/2373302736 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c02ccc0 con 0x7f0ca406caf0 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0ca37fe640 1 -- 172.21.15.70:0/2373302736 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c034610 con 0x7f0ca406caf0 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 -- 172.21.15.70:0/2373302736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406caf0 msgr2=0x7f0ca415e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 --2- 172.21.15.70:0/2373302736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406caf0 0x7f0ca415e0c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40739f0 tx=0x7f0c9c02bf80 comp rx=0 tx=0).stop 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 -- 172.21.15.70:0/2373302736 shutdown_connections 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 --2- 172.21.15.70:0/2373302736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406caf0 0x7f0ca415e0c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 --2- 172.21.15.70:0/2373302736 >> v2:172.21.15.96:3300/0 conn(0x7f0ca406e540 0x7f0ca409b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.413+0000 7f0caab32640 1 -- 172.21.15.70:0/2373302736 >> 172.21.15.70:0/2373302736 conn(0x7f0ca40b3f40 msgr2=0x7f0ca40b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.414+0000 7f0caab32640 1 -- 172.21.15.70:0/2373302736 shutdown_connections 2024-04-15T15:15:09.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.414+0000 7f0caab32640 1 -- 172.21.15.70:0/2373302736 wait complete. 2024-04-15T15:15:09.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.415+0000 7f0caab32640 1 Processor -- start 2024-04-15T15:15:09.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.415+0000 7f0caab32640 1 -- start start 2024-04-15T15:15:09.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca406caf0 0x7f0ca415bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 0x7f0ca40c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4071060 con 0x7f0ca406e920 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4070ee0 con 0x7f0ca406e540 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca409b8f0 con 0x7f0ca406caf0 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48112/0 (socket says 172.21.15.70:48112) 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2126917725 learned_addr learned my addr 172.21.15.70:0/2126917725 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca88ce640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca406caf0 0x7f0ca415bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca3fff640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 0x7f0ca40c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca406caf0 msgr2=0x7f0ca415bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca406caf0 0x7f0ca415bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 msgr2=0x7f0ca40c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 0x7f0ca40c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:09.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 -- 172.21.15.70:0/2126917725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca40c7130 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca3fff640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 0x7f0ca40c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca90cf640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c036040 tx=0x7f0c9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c043020 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c9c03e450 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca40c7410 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c9c03e820 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.416+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca40c79d0 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.417+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca40690f0 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.418+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c9c046020 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.418+0000 7f0ca17fa640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 0x7f0c8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.418+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7f0c9c0ae650 con 0x7f0ca406e920 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.418+0000 7f0ca88ce640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 0x7f0c8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:09.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.419+0000 7f0ca88ce640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 0x7f0c8806edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800aac0 tx=0x7f0c98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:09.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.423+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c9c04d020 con 0x7f0ca406e920 2024-04-15T15:15:09.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:09.637+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f0ca40ce880 con 0x7f0ca406e920 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.283+0000 7f0ca17fa640 1 -- 172.21.15.70:0/2126917725 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f0c9c07caf0 con 0x7f0ca406e920 2024-04-15T15:15:10.285 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 2 and data pool 3 2024-04-15T15:15:10.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.291+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 msgr2=0x7f0c8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.291+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 0x7f0c8806edb0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0c9800aac0 tx=0x7f0c98008040 comp rx=0 tx=0).stop 2024-04-15T15:15:10.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.291+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 msgr2=0x7f0ca40b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.291+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c036040 tx=0x7f0c9c004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 shutdown_connections 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c8f0 0x7f0c8806edb0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406e920 0x7f0ca40b6900 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca406e540 0x7f0ca40c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 --2- 172.21.15.70:0/2126917725 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca406caf0 0x7f0ca415bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.296+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 >> 172.21.15.70:0/2126917725 conn(0x7f0ca40b3f40 msgr2=0x7f0ca40af140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.299+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 shutdown_connections 2024-04-15T15:15:10.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.299+0000 7f0caab32640 1 -- 172.21.15.70:0/2126917725 wait complete. 2024-04-15T15:15:10.317 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-15T15:15:10.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-15T15:15:10.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.424+0000 7f576f31b640 1 Processor -- start 2024-04-15T15:15:10.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.424+0000 7f576f31b640 1 -- start start 2024-04-15T15:15:10.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576f31b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f57681623f0 0x7f57680d1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576f31b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57680d08b0 con 0x7f576809b3e0 2024-04-15T15:15:10.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576f31b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57680d0b90 con 0x7f5768162ee0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576f31b640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f57680d0680 con 0x7f57681623f0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576c8b6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f576809b3e0 0x7f57681627d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33438/0 (socket says 172.21.15.70:33438) 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576c8b6640 1 -- 172.21.15.70:0/1321336784 learned_addr learned my addr 172.21.15.70:0/1321336784 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1321336784 >> v2:172.21.15.96:3300/0 conn(0x7f57681623f0 0x7f57680d1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4277747946 0 0) 0x7f57680d08b0 con 0x7f576809b3e0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5754003520 con 0x7f576809b3e0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366038695 0 0) 0x7f5754003520 con 0x7f576809b3e0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 >> v2:172.21.15.96:3300/0 conn(0x7f57681623f0 msgr2=0x7f57680d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 --2- 172.21.15.70:0/1321336784 >> v2:172.21.15.96:3300/0 conn(0x7f57681623f0 0x7f57680d1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 >> v1:172.21.15.96:6789/0 conn(0x7f5768162ee0 legacy=0x7f576806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57680d0e70 con 0x7f576809b3e0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1321336784 >> v2:172.21.15.96:3300/0 conn(0x7f57681623f0 0x7f57680d1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f5758001670 con 0x7f576809b3e0 2024-04-15T15:15:10.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 >> v1:172.21.15.70:6789/0 conn(0x7f576809b3e0 legacy=0x7f57681627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57540039b0 0x7f5754003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 0x7f5754004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57540051c0 0x7f575400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f575400c030 con 0x7f5754004800 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f575400c1e0 con 0x7f57540039b0 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.425+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f575400c3e0 con 0x7f57540051c0 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 0x7f5754004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57540039b0 0x7f5754003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576d8b8640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57540051c0 0x7f575400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 -- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57540051c0 msgr2=0x7f575400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57540051c0 0x7f575400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 -- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57540039b0 msgr2=0x7f5754003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57540039b0 0x7f5754003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 -- 172.21.15.70:0/1321336784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5754003520 con 0x7f5754004800 2024-04-15T15:15:10.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f57540039b0 0x7f5754003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576d8b8640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57540051c0 0x7f575400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576d8b8640 1 -- 172.21.15.70:0/1321336784 reap_dead start 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.426+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 0x7f5754004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f575800c210 tx=0x7f5754005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c6e0e75a94d907 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758007d30 con 0x7f5754004800 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5758030070 con 0x7f5754004800 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f575ffff640 1 -- 172.21.15.70:0/1321336784 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f575802cda0 con 0x7f5754004800 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 -- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 msgr2=0x7f5754004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 0x7f5754004c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f575800c210 tx=0x7f5754005830 comp rx=0 tx=0).stop 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 -- 172.21.15.70:0/1321336784 shutdown_connections 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 --2- 172.21.15.70:0/1321336784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5754004800 0x7f5754004c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 -- 172.21.15.70:0/1321336784 >> 172.21.15.70:0/1321336784 conn(0x7f57680b3f70 msgr2=0x7f57680b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 -- 172.21.15.70:0/1321336784 shutdown_connections 2024-04-15T15:15:10.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.427+0000 7f576f31b640 1 -- 172.21.15.70:0/1321336784 wait complete. 2024-04-15T15:15:10.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.428+0000 7f576f31b640 1 Processor -- start 2024-04-15T15:15:10.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.428+0000 7f576f31b640 1 -- start start 2024-04-15T15:15:10.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5754004800 0x7f576815d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57680b55e0 0x7f57680b2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f57680d1670 con 0x7f576814c830 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f576809b840 con 0x7f5754004800 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576f31b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f57680d17f0 con 0x7f57680b55e0 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48124/0 (socket says 172.21.15.70:48124) 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 -- 172.21.15.70:0/1817664107 learned_addr learned my addr 172.21.15.70:0/1817664107 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576d8b8640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57680b55e0 0x7f57680b2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 -- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57680b55e0 msgr2=0x7f57680b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57680b55e0 0x7f57680b2dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 -- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5754004800 msgr2=0x7f576815d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.429+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5754004800 0x7f576815d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f576c8b6640 1 -- 172.21.15.70:0/1817664107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f576815ff30 con 0x7f576814c830 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f576c8b6640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f575800c210 tx=0x7f575802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758046070 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5758004bf0 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5768160210 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5758007d10 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.430+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57681607d0 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.431+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57680689e0 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.432+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5758007070 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.432+0000 7f575dffb640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 0x7f574c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.432+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f57580acfc0 con 0x7f576814c830 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.432+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 0x7f574c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:10.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.432+0000 7f576d0b7640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 0x7f574c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f57680b3f40 tx=0x7f5764008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:10.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.436+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f575807b370 con 0x7f576814c830 2024-04-15T15:15:10.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:10.625+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f57680b8400 con 0x7f576814c830 2024-04-15T15:15:11.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.287+0000 7f575dffb640 1 -- 172.21.15.70:0/1817664107 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f5758081020 con 0x7f576814c830 2024-04-15T15:15:11.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 msgr2=0x7f574c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 0x7f574c06edb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f57680b3f40 tx=0x7f5764008040 comp rx=0 tx=0).stop 2024-04-15T15:15:11.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 msgr2=0x7f57680b50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f575800c210 tx=0x7f575802e070 comp rx=0 tx=0).stop 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 shutdown_connections 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f57680b55e0 0x7f57680b2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f574c06c8f0 0x7f574c06edb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f576814c830 0x7f57680b50a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 --2- 172.21.15.70:0/1817664107 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5754004800 0x7f576815d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 >> 172.21.15.70:0/1817664107 conn(0x7f57680b3f70 msgr2=0x7f57680afab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 shutdown_connections 2024-04-15T15:15:11.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.290+0000 7f576f31b640 1 -- 172.21.15.70:0/1817664107 wait complete. 2024-04-15T15:15:11.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:15:11.360 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:11.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:11.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.420+0000 7f3d4a8b3640 1 Processor -- start 2024-04-15T15:15:11.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.420+0000 7f3d4a8b3640 1 -- start start 2024-04-15T15:15:11.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d4a8b3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440b3be0 0x7f3d440d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d4a8b3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d440d1610 0x7f3d440d2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d4a8b3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4406cbd0 con 0x7f3d4415ec00 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d4a8b3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3d4415efe0 con 0x7f3d440d1610 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d4a8b3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3d4406c980 con 0x7f3d440b3be0 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d440d1610 0x7f3d440d2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d43fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3d4415ec00 0x7f3d440cfac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:33440/0 (socket says 172.21.15.70:33440) 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d440d1610 0x7f3d440d2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48140/0 (socket says 172.21.15.70:48140) 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d43fff640 1 -- 172.21.15.70:0/1596571420 learned_addr learned my addr 172.21.15.70:0/1596571420 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d3bfff640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440b3be0 0x7f3d440d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 -- 172.21.15.70:0/1596571420 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440b3be0 msgr2=0x7f3d440d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440b3be0 0x7f3d440d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 -- 172.21.15.70:0/1596571420 >> v1:172.21.15.70:6789/0 conn(0x7f3d4415ec00 legacy=0x7f3d440cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d437fe640 1 -- 172.21.15.70:0/1596571420 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2166987344 0 0) 0x7f3d4406cbd0 con 0x7f3d4415ec00 2024-04-15T15:15:11.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.421+0000 7f3d48e50640 1 -- 172.21.15.70:0/1596571420 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4406ceb0 con 0x7f3d440d1610 2024-04-15T15:15:11.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.422+0000 7f3d48e50640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d440d1610 0x7f3d440d2a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009d90 tx=0x7f3d3c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd06389212c2c50e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.422+0000 7f3d437fe640 1 -- 172.21.15.70:0/1596571420 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c039070 con 0x7f3d440d1610 2024-04-15T15:15:11.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.422+0000 7f3d437fe640 1 -- 172.21.15.70:0/1596571420 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3c02cbc0 con 0x7f3d440d1610 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.422+0000 7f3d437fe640 1 -- 172.21.15.70:0/1596571420 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c034440 con 0x7f3d440d1610 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1596571420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d440d1610 msgr2=0x7f3d440d2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d440d1610 0x7f3d440d2a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009d90 tx=0x7f3d3c02be10 comp rx=0 tx=0).stop 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1596571420 shutdown_connections 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d440d1610 0x7f3d440d2a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1596571420 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440b3be0 0x7f3d440d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1596571420 >> 172.21.15.70:0/1596571420 conn(0x7f3d440b61d0 msgr2=0x7f3d440b65b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1596571420 shutdown_connections 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.423+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1596571420 wait complete. 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.424+0000 7f3d4a8b3640 1 Processor -- start 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.424+0000 7f3d4a8b3640 1 -- start start 2024-04-15T15:15:11.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d440b3be0 0x7f3d4414dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440d1610 0x7f3d441583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4406d990 con 0x7f3d4415ec00 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4406d470 con 0x7f3d440b3be0 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d4a8b3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3d440d2fa0 con 0x7f3d440d1610 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48150/0 (socket says 172.21.15.70:48150) 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 -- 172.21.15.70:0/1943092980 learned_addr learned my addr 172.21.15.70:0/1943092980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d3bfff640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440d1610 0x7f3d441583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d43fff640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d440b3be0 0x7f3d4414dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 -- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440d1610 msgr2=0x7f3d441583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440d1610 0x7f3d441583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 -- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d440b3be0 msgr2=0x7f3d4414dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d440b3be0 0x7f3d4414dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 -- 172.21.15.70:0/1943092980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d441618b0 con 0x7f3d4415ec00 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.425+0000 7f3d48e50640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009eb0 tx=0x7f3d3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.426+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c039050 con 0x7f3d4415ec00 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.426+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d3c045070 con 0x7f3d4415ec00 2024-04-15T15:15:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.426+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d440cc170 con 0x7f3d4415ec00 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.426+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3c009450 con 0x7f3d4415ec00 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.426+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d44160bc0 con 0x7f3d4415ec00 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.427+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d440689e0 con 0x7f3d4415ec00 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.427+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d3c03e850 con 0x7f3d4415ec00 2024-04-15T15:15:11.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.427+0000 7f3d417fa640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 0x7f3d2406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:11.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.428+0000 7f3d43fff640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 0x7f3d2406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:11.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.428+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7f3d3c0ae850 con 0x7f3d4415ec00 2024-04-15T15:15:11.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.428+0000 7f3d43fff640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 0x7f3d2406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d34007fd0 tx=0x7f3d34007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:11.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.432+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d3c07c620 con 0x7f3d4415ec00 2024-04-15T15:15:11.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:11.610+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f3d440b9620 con 0x7f3d4415ec00 2024-04-15T15:15:13.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.299+0000 7f3d417fa640 1 -- 172.21.15.70:0/1943092980 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7f3d3c080920 con 0x7f3d4415ec00 2024-04-15T15:15:13.301 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data_ec' created 2024-04-15T15:15:13.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 msgr2=0x7f3d2406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 0x7f3d2406edb0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d34007fd0 tx=0x7f3d34007de0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 msgr2=0x7f3d44158930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c009eb0 tx=0x7f3d3c004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 shutdown_connections 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d2406c8f0 0x7f3d2406edb0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d4415ec00 0x7f3d44158930 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d440d1610 0x7f3d441583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 --2- 172.21.15.70:0/1943092980 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d440b3be0 0x7f3d4414dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 >> 172.21.15.70:0/1943092980 conn(0x7f3d440b61d0 msgr2=0x7f3d44162ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.303+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 shutdown_connections 2024-04-15T15:15:13.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.304+0000 7f3d4a8b3640 1 -- 172.21.15.70:0/1943092980 wait complete. 2024-04-15T15:15:13.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-15T15:15:13.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.431+0000 7f8fcb55f640 1 Processor -- start 2024-04-15T15:15:13.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 -- start start 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 0x7f8fc40cee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8fc40cfbd0 0x7f8fc40c56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fc40cd110 con 0x7f8fc40b3a00 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc40cc870 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fcb55f640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8fc40ccb50 con 0x7f8fc415b660 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.432+0000 7f8fc9afc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8fc40cfbd0 0x7f8fc40c56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc8afa640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8fc40b3a00 0x7f8fc40cf4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50710/0 (socket says 172.21.15.70:50710) 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc8afa640 1 -- 172.21.15.70:0/2404710542 learned_addr learned my addr 172.21.15.70:0/2404710542 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc92fb640 1 --2- 172.21.15.70:0/2404710542 >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 0x7f8fc40cee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc9afc640 1 -- 172.21.15.70:0/2404710542 >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 msgr2=0x7f8fc40cee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc9afc640 1 --2- 172.21.15.70:0/2404710542 >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 0x7f8fc40cee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc9afc640 1 -- 172.21.15.70:0/2404710542 >> v1:172.21.15.70:6789/0 conn(0x7f8fc40b3a00 legacy=0x7f8fc40cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc3fff640 1 -- 172.21.15.70:0/2404710542 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3684484850 0 0) 0x7f8fc40cd110 con 0x7f8fc40b3a00 2024-04-15T15:15:13.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc9afc640 1 -- 172.21.15.70:0/2404710542 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc40cce30 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.433+0000 7f8fc92fb640 1 --2- 172.21.15.70:0/2404710542 >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 0x7f8fc40cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fc9afc640 1 --2- 172.21.15.70:0/2404710542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc40cfbd0 0x7f8fc40c56b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4071700 tx=0x7f8fbc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e95cbd97ec287d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fc3fff640 1 -- 172.21.15.70:0/2404710542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc034070 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fc3fff640 1 -- 172.21.15.70:0/2404710542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fbc030610 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fc3fff640 1 -- 172.21.15.70:0/2404710542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc0309e0 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 -- 172.21.15.70:0/2404710542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc40cfbd0 msgr2=0x7f8fc40c56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/2404710542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc40cfbd0 0x7f8fc40c56b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4071700 tx=0x7f8fbc02b960 comp rx=0 tx=0).stop 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 -- 172.21.15.70:0/2404710542 shutdown_connections 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/2404710542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc40cfbd0 0x7f8fc40c56b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/2404710542 >> v2:172.21.15.96:3300/0 conn(0x7f8fc415b660 0x7f8fc40cee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 -- 172.21.15.70:0/2404710542 >> 172.21.15.70:0/2404710542 conn(0x7f8fc40b2e70 msgr2=0x7f8fc40b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 -- 172.21.15.70:0/2404710542 shutdown_connections 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.434+0000 7f8fcb55f640 1 -- 172.21.15.70:0/2404710542 wait complete. 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 Processor -- start 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 -- start start 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc40b3a00 0x7f8fc4153710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8fc40cfbd0 0x7f8fc4159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc415ba40 con 0x7f8fc415b660 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8fc40c5190 con 0x7f8fc40b3a00 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fcb55f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8fc415bbc0 con 0x7f8fc40cfbd0 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fc9afc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fc9afc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54434/0 (socket says 172.21.15.70:54434) 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.436+0000 7f8fc9afc640 1 -- 172.21.15.70:0/3450763591 learned_addr learned my addr 172.21.15.70:0/3450763591 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc8afa640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8fc40cfbd0 0x7f8fc4159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc92fb640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc40b3a00 0x7f8fc4153710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 -- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8fc40cfbd0 msgr2=0x7f8fc4159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8fc40cfbd0 0x7f8fc4159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 -- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc40b3a00 msgr2=0x7f8fc4153710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc40b3a00 0x7f8fc4153710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 -- 172.21.15.70:0/3450763591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fc4157f70 con 0x7f8fc415b660 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc9afc640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009890 tx=0x7f8fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc034040 con 0x7f8fc415b660 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fbc004030 con 0x7f8fc415b660 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc004400 con 0x7f8fc415b660 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fc4158250 con 0x7f8fc415b660 2024-04-15T15:15:13.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.437+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fc4158810 con 0x7f8fc415b660 2024-04-15T15:15:13.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.438+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fc40690f0 con 0x7f8fc415b660 2024-04-15T15:15:13.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.439+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8fbc037070 con 0x7f8fc415b660 2024-04-15T15:15:13.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.439+0000 7f8fc1ffb640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 0x7f8fb006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:13.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.439+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f8fbc0adc00 con 0x7f8fc415b660 2024-04-15T15:15:13.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.439+0000 7f8fc92fb640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 0x7f8fb006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:13.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.440+0000 7f8fc92fb640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 0x7f8fb006edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4007510 tx=0x7f8fb40074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:13.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.444+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8fbc0b4080 con 0x7f8fc415b660 2024-04-15T15:15:13.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:13.618+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f8fc406f770 con 0x7f8fc415b660 2024-04-15T15:15:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.304+0000 7f8fc1ffb640 1 -- 172.21.15.70:0/3450763591 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f8fbc07be30 con 0x7f8fc415b660 2024-04-15T15:15:14.306 INFO:teuthology.orchestra.run.smithi070.stderr:set pool 4 allow_ec_overwrites to true 2024-04-15T15:15:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.307+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 msgr2=0x7f8fb006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.307+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 0x7f8fb006edb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4007510 tx=0x7f8fb40074a0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.307+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 msgr2=0x7f8fc40b6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.307+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc009890 tx=0x7f8fbc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 shutdown_connections 2024-04-15T15:15:14.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8fb006c8f0 0x7f8fb006edb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8fc415b660 0x7f8fc40b6340 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8fc40cfbd0 0x7f8fc4159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 --2- 172.21.15.70:0/3450763591 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8fc40b3a00 0x7f8fc4153710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 >> 172.21.15.70:0/3450763591 conn(0x7f8fc40b2e70 msgr2=0x7f8fc40c5900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 shutdown_connections 2024-04-15T15:15:14.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.308+0000 7f8fcb55f640 1 -- 172.21.15.70:0/3450763591 wait complete. 2024-04-15T15:15:14.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-15T15:15:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.440+0000 7fac0a132640 1 Processor -- start 2024-04-15T15:15:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.440+0000 7fac0a132640 1 -- start start 2024-04-15T15:15:14.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac0a132640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fac04070940 0x7fac0409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac0a132640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fac040d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac0a132640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac040d05e0 con 0x7fac04160310 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac0a132640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fac0415f720 con 0x7fac04070940 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac0a132640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fac0415fa00 con 0x7fac040d0da0 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac02ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fac04160310 0x7fac04070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50724/0 (socket says 172.21.15.70:50724) 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac02ffd640 1 -- 172.21.15.70:0/426367497 learned_addr learned my addr 172.21.15.70:0/426367497 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2444196564 0 0) 0x7fac040d05e0 con 0x7fac04160310 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabec003520 con 0x7fac04160310 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac03fff640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fac040d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> v2:172.21.15.96:3300/0 conn(0x7fac04070940 0x7fac0409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3428884004 0 0) 0x7fabec003520 con 0x7fac04160310 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 msgr2=0x7fac040d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fac040d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 >> v2:172.21.15.96:3300/0 conn(0x7fac04070940 msgr2=0x7fac0409b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 --2- 172.21.15.70:0/426367497 >> v2:172.21.15.96:3300/0 conn(0x7fac04070940 0x7fac0409b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac0415fce0 con 0x7fac04160310 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fabf8001670 con 0x7fac04160310 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> v2:172.21.15.96:3300/0 conn(0x7fac04070940 0x7fac0409b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 >> v1:172.21.15.70:6789/0 conn(0x7fac04160310 legacy=0x7fac04070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabec0039b0 0x7fabec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fabec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabec004d40 0x7fabec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac0415f720 con 0x7fabec0039b0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac0415fa00 con 0x7fabec004d40 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.441+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fabec00ba70 con 0x7fac040d0da0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabec0039b0 0x7fabec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac03fff640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabec004d40 0x7fabec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac02ffd640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fabec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 -- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 msgr2=0x7fabec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fabec004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 -- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabec004d40 msgr2=0x7fabec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fabec004d40 0x7fabec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 -- 172.21.15.70:0/426367497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec003520 con 0x7fabec0039b0 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac02ffd640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac040d0da0 0x7fabec004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.442+0000 7fac037fe640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabec0039b0 0x7fabec003dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabf40098d0 tx=0x7fabec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1007e0a62a3c3414 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf400f040 con 0x7fabec0039b0 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf4004230 con 0x7fabec0039b0 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac027fc640 1 -- 172.21.15.70:0/426367497 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf4003610 con 0x7fabec0039b0 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 -- 172.21.15.70:0/426367497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabec0039b0 msgr2=0x7fabec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 --2- 172.21.15.70:0/426367497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fabec0039b0 0x7fabec003dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fabf40098d0 tx=0x7fabec005830 comp rx=0 tx=0).stop 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac03fff640 1 -- 172.21.15.70:0/426367497 reap_dead start 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 -- 172.21.15.70:0/426367497 shutdown_connections 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 -- 172.21.15.70:0/426367497 >> 172.21.15.70:0/426367497 conn(0x7fac040b3f40 msgr2=0x7fac040b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 -- 172.21.15.70:0/426367497 shutdown_connections 2024-04-15T15:15:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.443+0000 7fac0a132640 1 -- 172.21.15.70:0/426367497 wait complete. 2024-04-15T15:15:14.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.444+0000 7fac0a132640 1 Processor -- start 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 -- start start 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fabec0039b0 0x7fac04153e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac040c8100 0x7fac040b6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac040d1230 con 0x7fac040c7c80 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac04160770 con 0x7fac040c8100 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac0a132640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fac040d13b0 con 0x7fabec0039b0 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac02ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac02ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54444/0 (socket says 172.21.15.70:54444) 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac02ffd640 1 -- 172.21.15.70:0/3518848970 learned_addr learned my addr 172.21.15.70:0/3518848970 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac037fe640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fabec0039b0 0x7fac04153e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.445+0000 7fac03fff640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac040c8100 0x7fac040b6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 -- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fabec0039b0 msgr2=0x7fac04153e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fabec0039b0 0x7fac04153e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 -- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac040c8100 msgr2=0x7fac040b6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac040c8100 0x7fac040b6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 -- 172.21.15.70:0/3518848970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac04154390 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac02ffd640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf800e840 tx=0x7fabf800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf800c850 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf8018070 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac040c69b0 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf800cbf0 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.446+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac040c6f70 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.447+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac040689e0 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.447+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fabf801d460 con 0x7fac040c7c80 2024-04-15T15:15:14.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.448+0000 7fabebfff640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 0x7fabf006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:14.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.448+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7fabf808bbc0 con 0x7fac040c7c80 2024-04-15T15:15:14.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.448+0000 7fac037fe640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 0x7fabf006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:14.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.448+0000 7fac037fe640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 0x7fabf006edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabf4035000 tx=0x7fabf4031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:14.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.452+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fac04154390 con 0x7fac040c7c80 2024-04-15T15:15:14.637 INFO:tasks.ceph.mds.c.smithi070.stderr:2024-04-15T15:15:14.635+0000 7f5b2561b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:15:14.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:14.671+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fac04070720 con 0x7fac040c7c80 2024-04-15T15:15:15.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.310+0000 7fabebfff640 1 -- 172.21.15.70:0/3518848970 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7fabf8059680 con 0x7fac040c7c80 2024-04-15T15:15:15.312 INFO:teuthology.orchestra.run.smithi070.stderr:added data pool 4 to fsmap 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 msgr2=0x7fabf006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 0x7fabf006edb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fabf4035000 tx=0x7fabf4031000 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 msgr2=0x7fac040b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabf800e840 tx=0x7fabf800ed00 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 shutdown_connections 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac040c8100 0x7fac040b6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabf006c8f0 0x7fabf006edb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac040c7c80 0x7fac040b6990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 --2- 172.21.15.70:0/3518848970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fabec0039b0 0x7fac04153e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 >> 172.21.15.70:0/3518848970 conn(0x7fac040b3f40 msgr2=0x7fac040b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 shutdown_connections 2024-04-15T15:15:15.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.313+0000 7fac0a132640 1 -- 172.21.15.70:0/3518848970 wait complete. 2024-04-15T15:15:15.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:15.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.429+0000 7f8cacd50640 1 Processor -- start 2024-04-15T15:15:15.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.429+0000 7f8cacd50640 1 -- start start 2024-04-15T15:15:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8cacd50640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8ca809a9d0 0x7f8ca8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8cacd50640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca8070720 con 0x7f8ca80701f0 2024-04-15T15:15:15.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8cacd50640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ca8070a90 con 0x7f8ca815fb30 2024-04-15T15:15:15.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8cacd50640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8ca809ba90 con 0x7f8ca809a9d0 2024-04-15T15:15:15.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8ca5d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8ca80701f0 0x7f8ca809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50738/0 (socket says 172.21.15.70:50738) 2024-04-15T15:15:15.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8ca5d9b640 1 -- 172.21.15.70:0/162835263 learned_addr learned my addr 172.21.15.70:0/162835263 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8ca659c640 1 --2- 172.21.15.70:0/162835263 >> v2:172.21.15.96:3300/0 conn(0x7f8ca809a9d0 0x7f8ca8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659409345 0 0) 0x7f8ca8070720 con 0x7f8ca80701f0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.430+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8c90003520 con 0x7f8ca80701f0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 771512894 0 0) 0x7f8c90003520 con 0x7f8ca80701f0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 >> v2:172.21.15.96:3300/0 conn(0x7f8ca809a9d0 msgr2=0x7f8ca8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 --2- 172.21.15.70:0/162835263 >> v2:172.21.15.96:3300/0 conn(0x7f8ca809a9d0 0x7f8ca8160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 >> v1:172.21.15.96:6789/0 conn(0x7f8ca815fb30 legacy=0x7f8ca80d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca8070d70 con 0x7f8ca80701f0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f8c9c001670 con 0x7f8ca80701f0 2024-04-15T15:15:15.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 >> v1:172.21.15.70:6789/0 conn(0x7f8ca80701f0 legacy=0x7f8ca809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8c90003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c90004800 0x7f8c90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c900051c0 0x7f8c9000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9000c030 con 0x7f8c900039b0 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8c9000c1e0 con 0x7f8c900051c0 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8c9000c3e0 con 0x7f8c90004800 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8c90003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca5d9b640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c90004800 0x7f8c90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca6d9d640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c900051c0 0x7f8c9000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 -- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c90004800 msgr2=0x7f8c90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c90004800 0x7f8c90004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 -- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c900051c0 msgr2=0x7f8c9000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c900051c0 0x7f8c9000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca659c640 1 -- 172.21.15.70:0/162835263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c90003520 con 0x7f8c900039b0 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca6d9d640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c900051c0 0x7f8c9000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.431+0000 7f8ca6d9d640 1 -- 172.21.15.70:0/162835263 reap_dead start 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8ca659c640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8c90003dd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009fc0 tx=0x7f8c90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98866dd9e114b703 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98039070 con 0x7f8c900039b0 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9802cc20 con 0x7f8c900039b0 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8ca559a640 1 -- 172.21.15.70:0/162835263 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98009c10 con 0x7f8c900039b0 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8cacd50640 1 -- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 msgr2=0x7f8c90003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.432+0000 7f8cacd50640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8c90003dd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009fc0 tx=0x7f8c90005830 comp rx=0 tx=0).stop 2024-04-15T15:15:15.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.433+0000 7f8cacd50640 1 -- 172.21.15.70:0/162835263 shutdown_connections 2024-04-15T15:15:15.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.433+0000 7f8cacd50640 1 --2- 172.21.15.70:0/162835263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8c90003dd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.433+0000 7f8cacd50640 1 -- 172.21.15.70:0/162835263 >> 172.21.15.70:0/162835263 conn(0x7f8ca80d2460 msgr2=0x7f8ca80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.433+0000 7f8cacd50640 1 -- 172.21.15.70:0/162835263 shutdown_connections 2024-04-15T15:15:15.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.433+0000 7f8cacd50640 1 -- 172.21.15.70:0/162835263 wait complete. 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.434+0000 7f8cacd50640 1 Processor -- start 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.434+0000 7f8cacd50640 1 -- start start 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ca814e040 0x7f8ca814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 0x7f8ca814d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca815ffc0 con 0x7f8c900039b0 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca80d08c0 con 0x7f8ca814ce10 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8ca80b1c60 con 0x7f8ca814e040 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54460/0 (socket says 172.21.15.70:54460) 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 -- 172.21.15.70:0/3357333151 learned_addr learned my addr 172.21.15.70:0/3357333151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca6d9d640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 0x7f8ca814d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca5d9b640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ca814e040 0x7f8ca814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 -- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ca814e040 msgr2=0x7f8ca814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ca814e040 0x7f8ca814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 -- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 msgr2=0x7f8ca814d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 0x7f8ca814d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 -- 172.21.15.70:0/3357333151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca814bca0 con 0x7f8c900039b0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca6d9d640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 0x7f8ca814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8ca659c640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800e230 tx=0x7f8c98009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98039050 con 0x7f8c900039b0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c9803a040 con 0x7f8c900039b0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca80c8190 con 0x7f8c900039b0 2024-04-15T15:15:15.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98009c10 con 0x7f8c900039b0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.435+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca814c330 con 0x7f8c900039b0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.436+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ca80689e0 con 0x7f8c900039b0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.437+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8c98003bb0 con 0x7f8c900039b0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.437+0000 7f8c96ffd640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 0x7f8c8806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.437+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8c980ae6d0 con 0x7f8c900039b0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.437+0000 7f8ca5d9b640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 0x7f8c8806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.438+0000 7f8ca5d9b640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 0x7f8c8806ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80750a0 tx=0x7f8c9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.442+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c9807c7c0 con 0x7f8c900039b0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.654+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8ca80c9260 con 0x7f8c900039b0 2024-04-15T15:15:15.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.655+0000 7f8c96ffd640 1 -- 172.21.15.70:0/3357333151 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5174 (secure 0 0 0) 0x7f8c9807c0c0 con 0x7f8c900039b0 2024-04-15T15:15:15.657 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:15.657 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:14.672611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:15.657 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 6 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 msgr2=0x7f8c8806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 0x7f8c8806ede0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80750a0 tx=0x7f8c9c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 msgr2=0x7f8ca814b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8c9800e230 tx=0x7f8c98009750 comp rx=0 tx=0).stop 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 shutdown_connections 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ca814ce10 0x7f8ca814d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ca814e040 0x7f8ca814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8c8806c920 0x7f8c8806ede0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 --2- 172.21.15.70:0/3357333151 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c900039b0 0x7f8ca814b760 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 >> 172.21.15.70:0/3357333151 conn(0x7f8ca80d2460 msgr2=0x7f8ca815f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 shutdown_connections 2024-04-15T15:15:15.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.657+0000 7f8cacd50640 1 -- 172.21.15.70:0/3357333151 wait complete. 2024-04-15T15:15:15.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.770+0000 7fe4bec60640 1 Processor -- start 2024-04-15T15:15:15.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.770+0000 7fe4bec60640 1 -- start start 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bec60640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 0x7fe4b8072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bec60640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b80b4190 con 0x7fe4b8072c60 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bec60640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b80b4470 con 0x7fe4b8158770 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bec60640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b80b4da0 con 0x7fe4b80b5230 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bd1fd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe4b8072c60 0x7fe4b806dd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50748/0 (socket says 172.21.15.70:50748) 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4bd1fd640 1 -- 172.21.15.70:0/367751372 learned_addr learned my addr 172.21.15.70:0/367751372 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4b7fff640 1 --2- 172.21.15.70:0/367751372 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 0x7fe4b8072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4b77fe640 1 -- 172.21.15.70:0/367751372 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1569128396 0 0) 0x7fe4b80b4190 con 0x7fe4b8072c60 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.771+0000 7fe4b77fe640 1 -- 172.21.15.70:0/367751372 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4a0004630 con 0x7fe4b8072c60 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.772+0000 7fe4b7fff640 1 -- 172.21.15.70:0/367751372 >> v1:172.21.15.96:6789/0 conn(0x7fe4b8158770 legacy=0x7fe4b815bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.772+0000 7fe4b7fff640 1 -- 172.21.15.70:0/367751372 >> v1:172.21.15.70:6789/0 conn(0x7fe4b8072c60 legacy=0x7fe4b806dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.772+0000 7fe4b7fff640 1 -- 172.21.15.70:0/367751372 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b80b4750 con 0x7fe4b80b5230 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.772+0000 7fe4b7fff640 1 --2- 172.21.15.70:0/367751372 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 0x7fe4b8072720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac0097d0 tx=0x7fe4ac02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8cc38aecc0c6846 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4b77fe640 1 -- 172.21.15.70:0/367751372 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac035020 con 0x7fe4b80b5230 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4b77fe640 1 -- 172.21.15.70:0/367751372 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4ac031890 con 0x7fe4b80b5230 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4b77fe640 1 -- 172.21.15.70:0/367751372 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac031c60 con 0x7fe4b80b5230 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 -- 172.21.15.70:0/367751372 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 msgr2=0x7fe4b8072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 --2- 172.21.15.70:0/367751372 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 0x7fe4b8072720 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac0097d0 tx=0x7fe4ac02b930 comp rx=0 tx=0).stop 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 -- 172.21.15.70:0/367751372 shutdown_connections 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 --2- 172.21.15.70:0/367751372 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b80b5230 0x7fe4b8072720 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 -- 172.21.15.70:0/367751372 >> 172.21.15.70:0/367751372 conn(0x7fe4b806be00 msgr2=0x7fe4b806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 -- 172.21.15.70:0/367751372 shutdown_connections 2024-04-15T15:15:15.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.773+0000 7fe4bec60640 1 -- 172.21.15.70:0/367751372 wait complete. 2024-04-15T15:15:15.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.774+0000 7fe4bec60640 1 Processor -- start 2024-04-15T15:15:15.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.774+0000 7fe4bec60640 1 -- start start 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b8072c60 0x7fe4b8151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4b80b5230 0x7fe4b81496a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b8158b50 con 0x7fe4b8158770 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b8071fa0 con 0x7fe4b80b5230 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bec60640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b80b3d20 con 0x7fe4b8072c60 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54476/0 (socket says 172.21.15.70:54476) 2024-04-15T15:15:15.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 -- 172.21.15.70:0/2420810259 learned_addr learned my addr 172.21.15.70:0/2420810259 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:15.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bc9fc640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b8072c60 0x7fe4b8151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4b7fff640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4b80b5230 0x7fe4b81496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 -- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b8072c60 msgr2=0x7fe4b8151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b8072c60 0x7fe4b8151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 -- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4b80b5230 msgr2=0x7fe4b81496a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.775+0000 7fe4bd1fd640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4b80b5230 0x7fe4b81496a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4bd1fd640 1 -- 172.21.15.70:0/2420810259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b80cc710 con 0x7fe4b8158770 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4bd1fd640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000cb40 tx=0x7fe4b000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0017070 con 0x7fe4b8158770 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b000aca0 con 0x7fe4b8158770 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0012620 con 0x7fe4b8158770 2024-04-15T15:15:15.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b80cc9b0 con 0x7fe4b8158770 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.776+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b81524b0 con 0x7fe4b8158770 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.777+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b80689e0 con 0x7fe4b8158770 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.777+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe4b001a070 con 0x7fe4b8158770 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.778+0000 7fe4b57fa640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 0x7fe4a406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.778+0000 7fe4bc9fc640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 0x7fe4a406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.778+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe4b008bcf0 con 0x7fe4b8158770 2024-04-15T15:15:15.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.778+0000 7fe4bc9fc640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 0x7fe4a406edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8007fd0 tx=0x7fe4a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:15.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.783+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe4b0090070 con 0x7fe4b8158770 2024-04-15T15:15:15.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.959+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4b8069140 con 0x7fe4b8158770 2024-04-15T15:15:15.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.960+0000 7fe4b57fa640 1 -- 172.21.15.70:0/2420810259 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7fe4b0059de0 con 0x7fe4b8158770 2024-04-15T15:15:15.962 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:15.963 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:15.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 msgr2=0x7fe4a406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 0x7fe4a406edb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8007fd0 tx=0x7fe4a8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:15.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 msgr2=0x7fe4b80b1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000cb40 tx=0x7fe4b000a060 comp rx=0 tx=0).stop 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 shutdown_connections 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe4a406c8f0 0x7fe4a406edb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe4b8158770 0x7fe4b80b1200 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe4b80b5230 0x7fe4b81496a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 --2- 172.21.15.70:0/2420810259 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe4b8072c60 0x7fe4b8151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.962+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 >> 172.21.15.70:0/2420810259 conn(0x7fe4b806be00 msgr2=0x7fe4b814f940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.963+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 shutdown_connections 2024-04-15T15:15:15.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:15.963+0000 7fe4bec60640 1 -- 172.21.15.70:0/2420810259 wait complete. 2024-04-15T15:15:15.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:16.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 Processor -- start 2024-04-15T15:15:16.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 -- start start 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8da80cd000 0x7f8da80d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da815dbd0 con 0x7f8da809ab40 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da80cd6a0 con 0x7f8da81602d0 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8dadceb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8da80cd460 con 0x7f8da80cd000 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8da6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8da80cd000 0x7f8da80d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.078+0000 7f8da7fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8da809ab40 0x7f8da809afd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50764/0 (socket says 172.21.15.70:50764) 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da7fff640 1 -- 172.21.15.70:0/3572927318 learned_addr learned my addr 172.21.15.70:0/3572927318 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da6ffd640 1 -- 172.21.15.70:0/3572927318 >> v1:172.21.15.96:6789/0 conn(0x7f8da81602d0 legacy=0x7f8da81606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da6ffd640 1 -- 172.21.15.70:0/3572927318 >> v1:172.21.15.70:6789/0 conn(0x7f8da809ab40 legacy=0x7f8da809afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da6ffd640 1 -- 172.21.15.70:0/3572927318 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da80cd980 con 0x7f8da80cd000 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da67fc640 1 -- 172.21.15.70:0/3572927318 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992950414 0 0) 0x7f8da815dbd0 con 0x7f8da809ab40 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.079+0000 7f8da6ffd640 1 --2- 172.21.15.70:0/3572927318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da80cd000 0x7f8da80d0ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0097a0 tx=0x7f8d9c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef10880a906879e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8da67fc640 1 -- 172.21.15.70:0/3572927318 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c034070 con 0x7f8da80cd000 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8da67fc640 1 -- 172.21.15.70:0/3572927318 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9c030610 con 0x7f8da80cd000 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8da67fc640 1 -- 172.21.15.70:0/3572927318 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c0309e0 con 0x7f8da80cd000 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 -- 172.21.15.70:0/3572927318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da80cd000 msgr2=0x7f8da80d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 --2- 172.21.15.70:0/3572927318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da80cd000 0x7f8da80d0ef0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0097a0 tx=0x7f8d9c02b960 comp rx=0 tx=0).stop 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 -- 172.21.15.70:0/3572927318 shutdown_connections 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 --2- 172.21.15.70:0/3572927318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da80cd000 0x7f8da80d0ef0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 -- 172.21.15.70:0/3572927318 >> 172.21.15.70:0/3572927318 conn(0x7f8da80d2460 msgr2=0x7f8da80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 -- 172.21.15.70:0/3572927318 shutdown_connections 2024-04-15T15:15:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.080+0000 7f8dadceb640 1 -- 172.21.15.70:0/3572927318 wait complete. 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.081+0000 7f8dadceb640 1 Processor -- start 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 -- start start 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8da80cd000 0x7f8da80c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8da81602d0 0x7f8da80b7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80ce220 con 0x7f8da809ab40 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8da80cdf40 con 0x7f8da80cd000 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8dadceb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8da80d1430 con 0x7f8da81602d0 2024-04-15T15:15:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8da77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8da77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54500/0 (socket says 172.21.15.70:54500) 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.082+0000 7f8da77fe640 1 -- 172.21.15.70:0/2115630890 learned_addr learned my addr 172.21.15.70:0/2115630890 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da7fff640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8da81602d0 0x7f8da80b7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 -- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8da81602d0 msgr2=0x7f8da80b7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8da81602d0 0x7f8da80b7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da6ffd640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8da80cd000 0x7f8da80c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 -- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8da80cd000 msgr2=0x7f8da80c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8da80cd000 0x7f8da80c4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 -- 172.21.15.70:0/2115630890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da80b5220 con 0x7f8da809ab40 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8da77fe640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c400 tx=0x7f8d9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98017070 con 0x7f8da809ab40 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d980052d0 con 0x7f8da809ab40 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da80b54c0 con 0x7f8da809ab40 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98003830 con 0x7f8da809ab40 2024-04-15T15:15:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.083+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da80b5a80 con 0x7f8da809ab40 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.084+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da80689e0 con 0x7f8da809ab40 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.085+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d9801a070 con 0x7f8da809ab40 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.085+0000 7f8d8ffff640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 0x7f8d9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.085+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8d9808b140 con 0x7f8da809ab40 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.085+0000 7f8da6ffd640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 0x7f8d9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.085+0000 7f8da6ffd640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 0x7f8d9406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c036000 tx=0x7f8d9c03b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.089+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d980592c0 con 0x7f8da809ab40 2024-04-15T15:15:16.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.265+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8da80b4e20 con 0x7f8da809ab40 2024-04-15T15:15:16.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.267+0000 7f8d8ffff640 1 -- 172.21.15.70:0/2115630890 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f8d98058bc0 con 0x7f8da809ab40 2024-04-15T15:15:16.269 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:16.269 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:16.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.268+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 msgr2=0x7f8d9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.268+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 0x7f8d9406edb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c036000 tx=0x7f8d9c03b040 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.268+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 msgr2=0x7f8da8155930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.268+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8d9800c400 tx=0x7f8d9800c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 shutdown_connections 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8da81602d0 0x7f8da80b7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8da80cd000 0x7f8da80c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d9406c8f0 0x7f8d9406edb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 --2- 172.21.15.70:0/2115630890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8da809ab40 0x7f8da8155930 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 >> 172.21.15.70:0/2115630890 conn(0x7f8da80d2460 msgr2=0x7f8da80718a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 shutdown_connections 2024-04-15T15:15:16.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.269+0000 7f8dadceb640 1 -- 172.21.15.70:0/2115630890 wait complete. 2024-04-15T15:15:16.297 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:16.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.398+0000 7efd778e0640 1 Processor -- start 2024-04-15T15:15:16.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.398+0000 7efd778e0640 1 -- start start 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd778e0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd700b0560 0x7efd7006b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd778e0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700cbd10 con 0x7efd700b09a0 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd778e0640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd700cbff0 con 0x7efd7015eb00 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd778e0640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efd7015eee0 con 0x7efd700b0560 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd75e7d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efd700b09a0 0x7efd700ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50778/0 (socket says 172.21.15.70:50778) 2024-04-15T15:15:16.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.399+0000 7efd75e7d640 1 -- 172.21.15.70:0/2782749922 learned_addr learned my addr 172.21.15.70:0/2782749922 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:16.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3779301451 0 0) 0x7efd700cbd10 con 0x7efd700b09a0 2024-04-15T15:15:16.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd74e7b640 1 --2- 172.21.15.70:0/2782749922 >> v2:172.21.15.96:3300/0 conn(0x7efd700b0560 0x7efd7006b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd58003520 con 0x7efd700b09a0 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2301068774 0 0) 0x7efd58003520 con 0x7efd700b09a0 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 >> v2:172.21.15.96:3300/0 conn(0x7efd700b0560 msgr2=0x7efd7006b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 --2- 172.21.15.70:0/2782749922 >> v2:172.21.15.96:3300/0 conn(0x7efd700b0560 0x7efd7006b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 >> v1:172.21.15.96:6789/0 conn(0x7efd7015eb00 legacy=0x7efd700ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd700cc2d0 con 0x7efd700b09a0 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7efd68001670 con 0x7efd700b09a0 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 >> v1:172.21.15.70:6789/0 conn(0x7efd700b09a0 legacy=0x7efd700ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd58004800 0x7efd58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd580051c0 0x7efd5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd7015eee0 con 0x7efd580039b0 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd5800c060 con 0x7efd580051c0 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efd5800c260 con 0x7efd58004800 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd75e7d640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd580051c0 0x7efd5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd74e7b640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd58004800 0x7efd58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 -- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd58004800 msgr2=0x7efd58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd58004800 0x7efd58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 -- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd580051c0 msgr2=0x7efd5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd580051c0 0x7efd5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd7567c640 1 -- 172.21.15.70:0/2782749922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd58003520 con 0x7efd580039b0 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd75e7d640 1 -- 172.21.15.70:0/2782749922 reap_dead start 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.400+0000 7efd74e7b640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd58004800 0x7efd58004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.401+0000 7efd7567c640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd58003dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd60001e50 tx=0x7efd58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d393cb2cc9b42b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.401+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60001a70 con 0x7efd580039b0 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.401+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60003b70 con 0x7efd580039b0 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.401+0000 7efd6ffff640 1 -- 172.21.15.70:0/2782749922 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd600050d0 con 0x7efd580039b0 2024-04-15T15:15:16.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 -- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 msgr2=0x7efd58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd58003dd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd60001e50 tx=0x7efd58005830 comp rx=0 tx=0).stop 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 -- 172.21.15.70:0/2782749922 shutdown_connections 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 --2- 172.21.15.70:0/2782749922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd58003dd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 -- 172.21.15.70:0/2782749922 >> 172.21.15.70:0/2782749922 conn(0x7efd7009a9d0 msgr2=0x7efd700d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 -- 172.21.15.70:0/2782749922 shutdown_connections 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.402+0000 7efd778e0640 1 -- 172.21.15.70:0/2782749922 wait complete. 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.403+0000 7efd778e0640 1 Processor -- start 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.403+0000 7efd778e0640 1 -- start start 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 0x7efd700c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 0x7efd700c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd700cec60 con 0x7efd580039b0 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd700cc890 con 0x7efd700c5080 2024-04-15T15:15:16.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efd700cede0 con 0x7efd700c5a40 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54518/0 (socket says 172.21.15.70:54518) 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 -- 172.21.15.70:0/1814538570 learned_addr learned my addr 172.21.15.70:0/1814538570 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd75e7d640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 0x7efd700c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 -- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 msgr2=0x7efd700c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd74e7b640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 0x7efd700c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 0x7efd700c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 -- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 msgr2=0x7efd700c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 0x7efd700c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 -- 172.21.15.70:0/1814538570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd701f4920 con 0x7efd580039b0 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd75e7d640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 0x7efd700c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd74e7b640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 0x7efd700c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd7567c640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd60000f80 tx=0x7efd600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60003970 con 0x7efd580039b0 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd60006b50 con 0x7efd580039b0 2024-04-15T15:15:16.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701f4c00 con 0x7efd580039b0 2024-04-15T15:15:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.404+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd60007020 con 0x7efd580039b0 2024-04-15T15:15:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.405+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd701f51c0 con 0x7efd580039b0 2024-04-15T15:15:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.406+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd70070650 con 0x7efd580039b0 2024-04-15T15:15:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.406+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efd60049060 con 0x7efd580039b0 2024-04-15T15:15:16.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.406+0000 7efd6dffb640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 0x7efd5c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.407+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efd600ad300 con 0x7efd580039b0 2024-04-15T15:15:16.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.407+0000 7efd74e7b640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 0x7efd5c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.407+0000 7efd74e7b640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 0x7efd5c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd700b3910 tx=0x7efd64008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.411+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd70070650 con 0x7efd580039b0 2024-04-15T15:15:16.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.624+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efd700690f0 con 0x7efd580039b0 2024-04-15T15:15:16.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.624+0000 7efd6dffb640 1 -- 172.21.15.70:0/1814538570 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5174 (secure 0 0 0) 0x7efd60033020 con 0x7efd580039b0 2024-04-15T15:15:16.626 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:16.627 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:14.672611+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:16.627 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 6 2024-04-15T15:15:16.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 msgr2=0x7efd5c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 0x7efd5c06edb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7efd700b3910 tx=0x7efd64008040 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 msgr2=0x7efd70155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efd60000f80 tx=0x7efd600014a0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 shutdown_connections 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd700c5a40 0x7efd700c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd700c5080 0x7efd700c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd5c06c8f0 0x7efd5c06edb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 --2- 172.21.15.70:0/1814538570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd580039b0 0x7efd70155c00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 >> 172.21.15.70:0/1814538570 conn(0x7efd7009a9d0 msgr2=0x7efd7006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 shutdown_connections 2024-04-15T15:15:16.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.626+0000 7efd778e0640 1 -- 172.21.15.70:0/1814538570 wait complete. 2024-04-15T15:15:16.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:16.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.740+0000 7f75abfff640 1 Processor -- start 2024-04-15T15:15:16.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.740+0000 7f75abfff640 1 -- start start 2024-04-15T15:15:16.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75abfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75abfff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f75ac0b4590 0x7f75ac0b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75abfff640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ac160a80 con 0x7f75ac0d1a30 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75abfff640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f75ac15d920 con 0x7f75ac0b4590 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75abfff640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f75ac160d60 con 0x7f75ac15d500 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f75ac0b4590 0x7f75ac0b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75a27fc640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75a27fc640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43342/0 (socket says 172.21.15.70:43342) 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f75ac0b4590 0x7f75ac0b62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:33704/0 (socket says 172.21.15.70:33704) 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75a27fc640 1 -- 172.21.15.70:0/3024367278 learned_addr learned my addr 172.21.15.70:0/3024367278 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75aaffd640 1 --1- 172.21.15.70:0/3024367278 >> v1:172.21.15.96:6789/0 conn(0x7f75ac0d1a30 0x7f75ac15ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:60400/0 (socket says 172.21.15.70:60400) 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 -- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 msgr2=0x7f75ac0b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 -- 172.21.15.70:0/3024367278 >> v1:172.21.15.96:6789/0 conn(0x7f75ac0d1a30 legacy=0x7f75ac15ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75aa7fc640 1 -- 172.21.15.70:0/3024367278 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2766235974 0 0) 0x7f75ac160a80 con 0x7f75ac0d1a30 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75ab7fe640 1 -- 172.21.15.70:0/3024367278 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac15cd00 con 0x7f75ac0b4590 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.741+0000 7f75a27fc640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:16.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.742+0000 7f75ab7fe640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0b4590 0x7f75ac0b62a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009d90 tx=0x7f75a402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c7ab926284ffdf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.742+0000 7f75aa7fc640 1 -- 172.21.15.70:0/3024367278 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4039070 con 0x7f75ac0b4590 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.742+0000 7f75aa7fc640 1 -- 172.21.15.70:0/3024367278 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a402cc50 con 0x7f75ac0b4590 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.742+0000 7f75aa7fc640 1 -- 172.21.15.70:0/3024367278 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4034440 con 0x7f75ac0b4590 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 -- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0b4590 msgr2=0x7f75ac0b62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0b4590 0x7f75ac0b62a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009d90 tx=0x7f75a402bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 -- 172.21.15.70:0/3024367278 shutdown_connections 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0b4590 0x7f75ac0b62a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 --2- 172.21.15.70:0/3024367278 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac15d500 0x7f75ac0b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 -- 172.21.15.70:0/3024367278 >> 172.21.15.70:0/3024367278 conn(0x7f75ac06bbc0 msgr2=0x7f75ac06bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 -- 172.21.15.70:0/3024367278 shutdown_connections 2024-04-15T15:15:16.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.743+0000 7f75abfff640 1 -- 172.21.15.70:0/3024367278 wait complete. 2024-04-15T15:15:16.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.744+0000 7f75abfff640 1 Processor -- start 2024-04-15T15:15:16.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.744+0000 7f75abfff640 1 -- start start 2024-04-15T15:15:16.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac0b4590 0x7f75ac13c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0d1a30 0x7f75ac0b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac071a70 con 0x7f75ac15d500 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f75ac15c9f0 con 0x7f75ac0d1a30 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75abfff640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f75ac1606f0 con 0x7f75ac0b4590 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54534/0 (socket says 172.21.15.70:54534) 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 -- 172.21.15.70:0/2041894287 learned_addr learned my addr 172.21.15.70:0/2041894287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75a27fc640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0d1a30 0x7f75ac0b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75aaffd640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac0b4590 0x7f75ac13c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 -- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac0b4590 msgr2=0x7f75ac13c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac0b4590 0x7f75ac13c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 -- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0d1a30 msgr2=0x7f75ac0b21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0d1a30 0x7f75ac0b21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 -- 172.21.15.70:0/2041894287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac15c120 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75ab7fe640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009d90 tx=0x7f75a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4044020 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a403a040 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.745+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4039050 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.746+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ac06cc20 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.746+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ac13c950 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.747+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75ac04fd50 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.747+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f75a4003990 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.747+0000 7f75a8ff9640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 0x7f758c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.748+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f75a40adeb0 con 0x7f75ac15d500 2024-04-15T15:15:16.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.748+0000 7f75aaffd640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 0x7f758c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:16.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.748+0000 7f75aaffd640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 0x7f758c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f759c007f70 tx=0x7f759c007f00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:16.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.752+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f75a407b960 con 0x7f75ac15d500 2024-04-15T15:15:16.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.928+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75ac153860 con 0x7f75ac15d500 2024-04-15T15:15:16.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.929+0000 7f75a8ff9640 1 -- 172.21.15.70:0/2041894287 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f75a407fc60 con 0x7f75ac15d500 2024-04-15T15:15:16.931 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:16.932 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 msgr2=0x7f758c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 0x7f758c06edb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f759c007f70 tx=0x7f759c007f00 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 msgr2=0x7f75ac0b2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f75a4009d90 tx=0x7f75a4004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 shutdown_connections 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f758c06c8f0 0x7f758c06edb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f75ac15d500 0x7f75ac0b2720 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f75ac0d1a30 0x7f75ac0b21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 --2- 172.21.15.70:0/2041894287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f75ac0b4590 0x7f75ac13c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.931+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 >> 172.21.15.70:0/2041894287 conn(0x7f75ac06bbc0 msgr2=0x7f75ac06bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.932+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 shutdown_connections 2024-04-15T15:15:16.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:16.932+0000 7f75abfff640 1 -- 172.21.15.70:0/2041894287 wait complete. 2024-04-15T15:15:17.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:15:09.639145+0000', 'modified': '2024-04-15T15:15:14.672611+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3048652815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6835', 'nonce': 3048652815}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3048652815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-04-15T15:15:17.348 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:15:17.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:15:17.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.453+0000 7fe377f40640 1 Processor -- start 2024-04-15T15:15:17.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.453+0000 7fe377f40640 1 -- start start 2024-04-15T15:15:17.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe377f40640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe37015e790 0x7fe3700b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe377f40640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe3700b7be0 0x7fe3700d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe377f40640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe37015eb70 con 0x7fe3700d1b20 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe377f40640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe3700d3790 con 0x7fe37015e790 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe377f40640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe3700d45b0 con 0x7fe3700b7be0 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe375cdc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe37015e790 0x7fe3700b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe3764dd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe3700d1b20 0x7fe3700d4a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50780/0 (socket says 172.21.15.70:50780) 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe3764dd640 1 -- 172.21.15.70:0/4270537689 learned_addr learned my addr 172.21.15.70:0/4270537689 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.454+0000 7fe3754db640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe3700b7be0 0x7fe3700d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe375cdc640 1 -- 172.21.15.70:0/4270537689 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe3700b7be0 msgr2=0x7fe3700d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe375cdc640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe3700b7be0 0x7fe3700d15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe375cdc640 1 -- 172.21.15.70:0/4270537689 >> v1:172.21.15.70:6789/0 conn(0x7fe3700d1b20 legacy=0x7fe3700d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe375cdc640 1 -- 172.21.15.70:0/4270537689 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3700d3c80 con 0x7fe37015e790 2024-04-15T15:15:17.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe374cda640 1 -- 172.21.15.70:0/4270537689 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4226955021 0 0) 0x7fe37015eb70 con 0x7fe3700d1b20 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe375cdc640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37015e790 0x7fe3700b76a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0097a0 tx=0x7fe36c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b710d220870b0c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe374cda640 1 -- 172.21.15.70:0/4270537689 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c038070 con 0x7fe37015e790 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe374cda640 1 -- 172.21.15.70:0/4270537689 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c031700 con 0x7fe37015e790 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.455+0000 7fe374cda640 1 -- 172.21.15.70:0/4270537689 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c031ad0 con 0x7fe37015e790 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 -- 172.21.15.70:0/4270537689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37015e790 msgr2=0x7fe3700b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37015e790 0x7fe3700b76a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0097a0 tx=0x7fe36c02b7a0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 -- 172.21.15.70:0/4270537689 shutdown_connections 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe3700b7be0 0x7fe3700d15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 --2- 172.21.15.70:0/4270537689 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe37015e790 0x7fe3700b76a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 -- 172.21.15.70:0/4270537689 >> 172.21.15.70:0/4270537689 conn(0x7fe370067b40 msgr2=0x7fe370067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 -- 172.21.15.70:0/4270537689 shutdown_connections 2024-04-15T15:15:17.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.456+0000 7fe377f40640 1 -- 172.21.15.70:0/4270537689 wait complete. 2024-04-15T15:15:17.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.457+0000 7fe377f40640 1 Processor -- start 2024-04-15T15:15:17.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.457+0000 7fe377f40640 1 -- start start 2024-04-15T15:15:17.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 0x7fe3700cae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 0x7fe3700cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700d5150 con 0x7fe3700b7be0 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe3700d4240 con 0x7fe3700d1b20 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe377f40640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe3700d52d0 con 0x7fe37015e790 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54554/0 (socket says 172.21.15.70:54554) 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 -- 172.21.15.70:0/902923066 learned_addr learned my addr 172.21.15.70:0/902923066 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe3764dd640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 0x7fe3700cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe3754db640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 0x7fe3700cae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 -- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 msgr2=0x7fe3700cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 0x7fe3700cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 -- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 msgr2=0x7fe3700cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 0x7fe3700cae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 -- 172.21.15.70:0/902923066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3700cb3c0 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe3764dd640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 0x7fe3700cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe3754db640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 0x7fe3700cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.458+0000 7fe375cdc640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe36c036040 tx=0x7fe36c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.459+0000 7fe3667fc640 1 -- 172.21.15.70:0/902923066 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c042070 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.459+0000 7fe3667fc640 1 -- 172.21.15.70:0/902923066 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c033a30 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.459+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3700ab3a0 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.459+0000 7fe3667fc640 1 -- 172.21.15.70:0/902923066 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c038040 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.459+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3700ab960 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.460+0000 7fe3667fc640 1 -- 172.21.15.70:0/902923066 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe36c03b070 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.461+0000 7fe3667fc640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 0x7fe35406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.461+0000 7fe3754db640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 0x7fe35406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.461+0000 7fe3667fc640 1 -- 172.21.15.70:0/902923066 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe36c0ad840 con 0x7fe3700b7be0 2024-04-15T15:15:17.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.461+0000 7fe3754db640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 0x7fe35406edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe3700b5ab0 tx=0x7fe360008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 Processor -- start 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 -- start start 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a96ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 0x562ff5a970e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ff59d19c0 con 0x562ff5a93e10 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562ff5978650 con 0x562ff5a941f0 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562ff59cd980 con 0x562ff5a672f0 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54562/0 (socket says 172.21.15.70:54562) 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.462+0000 7fe375cdc640 1 -- 172.21.15.70:0/1562543738 learned_addr learned my addr 172.21.15.70:0/1562543738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3764dd640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 0x562ff5a970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3754db640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a96ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 -- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 msgr2=0x562ff5a970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 0x562ff5a970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 -- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 msgr2=0x562ff5a96ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a96ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 -- 172.21.15.70:0/1562543738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ff56e42d0 con 0x562ff5a93e10 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3764dd640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 0x562ff5a970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3754db640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a96ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe375cdc640 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0ab9a0 tx=0x7fe36c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e847ba0fdc58bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3537fe640 1 -- 172.21.15.70:0/1562543738 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c04b070 con 0x562ff5a93e10 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3537fe640 1 -- 172.21.15.70:0/1562543738 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c0abe50 con 0x562ff5a93e10 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.463+0000 7fe3537fe640 1 -- 172.21.15.70:0/1562543738 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c07ba50 con 0x562ff5a93e10 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- 172.21.15.70:0/1562543738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 msgr2=0x562ff5a66db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe36c0ab9a0 tx=0x7fe36c009190 comp rx=0 tx=0).stop 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- 172.21.15.70:0/1562543738 shutdown_connections 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a672f0 0x562ff5a970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a96ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- 172.21.15.70:0/1562543738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a66db0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- 172.21.15.70:0/1562543738 >> 172.21.15.70:0/1562543738 conn(0x562ff5a69020 msgr2=0x562ff5ab1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- 172.21.15.70:0/1562543738 shutdown_connections 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- 172.21.15.70:0/1562543738 wait complete. 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- start start 2024-04-15T15:15:17.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 Processor -- start 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 0x562ff5aac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 0x562ff5aaa700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ff59d19c0 con 0x562ff5a941f0 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562ff598ba10 con 0x562ff5a9a120 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562ff5a6b5c0 con 0x562ff5a93e10 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe37ac06740 10 client.0 _ll_register_callbacks cb 0x562ff59c7250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe3754db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe3754db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54578/0 (socket says 172.21.15.70:54578) 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe3754db640 1 -- 172.21.15.70:0/132477521 learned_addr learned my addr 172.21.15.70:0/132477521 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe3764dd640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 0x562ff5aaa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.464+0000 7fe375cdc640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 0x562ff5aac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 -- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 msgr2=0x562ff5aac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 0x562ff5aac5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 -- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 msgr2=0x562ff5aaa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 0x562ff5aaa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 -- 172.21.15.70:0/132477521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ff57ae6b0 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3764dd640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 0x562ff5aaa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe375cdc640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 0x562ff5aac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe3754db640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe360014460 tx=0x7fe360004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f5e622b5da8687 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe360015820 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe3600159c0 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe36c0abe50 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fe360015bc0 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 10 client.4452 fetch_fsmap learned FSMap version 6 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x562ff5aacb10 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c0abe50 con 0x562ff5a941f0 2024-04-15T15:15:17.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fe360031080 con 0x562ff5a941f0 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 10 client.4452 fetch_fsmap finished waiting for FSMap version 6 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 10 client.4452 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 20 client.4452 populate_metadata read hostname 'smithi070' 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5acdb60 0x562ff5acff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 4 client.4452 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 --> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562ff59df3e0 con 0x562ff5acdb60 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.465+0000 7fe375cdc640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5acdb60 0x562ff5acff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe350ff9640 10 client.4452 ms_handle_connect on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe375cdc640 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5acdb60 0x562ff5acff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a879bb24fe11f07 server_cookie=308907482656a7e5 in_seq=0 out_seq=0 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe350ff9640 1 -- 172.21.15.70:0/132477521 <== mds.0 v2:172.21.15.70:6835/3048652815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fe36c08aae0 con 0x562ff5acdb60 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe350ff9640 10 client.4452 handle_command_reply: tid=0 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 client.4452 shutdown 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 msgr2=0x562ff5a99be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe360014460 tx=0x7fe360004730 comp rx=0 tx=0).stop 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 shutdown_connections 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5acdb60 0x562ff5acff50 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a9a120 0x562ff5aaa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a941f0 0x562ff5a99be0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe350ff9640 0 client.4452 ms_handle_reset on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.466+0000 7fe37ac06740 1 --2- 172.21.15.70:0/132477521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a93e10 0x562ff5aac5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.467+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 >> 172.21.15.70:0/132477521 conn(0x562ff5a672f0 msgr2=0x562ff5a96b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.467+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 shutdown_connections 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.467+0000 7fe37ac06740 1 -- 172.21.15.70:0/132477521 wait complete. 2024-04-15T15:15:17.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.467+0000 7fe37ac06740 20 client.4452 trim_cache size 0 max 16384 2024-04-15T15:15:17.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 Processor -- start 2024-04-15T15:15:17.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 -- start start 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a968c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5a96e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe370202150 con 0x562ff5a93e10 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe370201130 con 0x562ff5a941f0 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.485+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe370200480 con 0x562ff5a9a120 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54588/0 (socket says 172.21.15.70:54588) 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 -- 172.21.15.70:0/2492751073 learned_addr learned my addr 172.21.15.70:0/2492751073 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 -- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 msgr2=0x562ff5a96e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5a96e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 -- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 msgr2=0x562ff5a968c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 -- 172.21.15.70:0/2492751073 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562ff5acbfc0 con 0x562ff5a93e10 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe375cdc640 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe36c07f040 tx=0x7fe36c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c01414bbce317 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe352ffd640 1 -- 172.21.15.70:0/2492751073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c031700 con 0x562ff5a93e10 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe352ffd640 1 -- 172.21.15.70:0/2492751073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c085070 con 0x562ff5a93e10 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.486+0000 7fe352ffd640 1 -- 172.21.15.70:0/2492751073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c07f700 con 0x562ff5a93e10 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- 172.21.15.70:0/2492751073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 msgr2=0x562ff5ad57d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe36c07f040 tx=0x7fe36c048000 comp rx=0 tx=0).stop 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- 172.21.15.70:0/2492751073 shutdown_connections 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5a96e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5a968c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 --2- 172.21.15.70:0/2492751073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5ad57d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- 172.21.15.70:0/2492751073 >> 172.21.15.70:0/2492751073 conn(0x562ff5a672f0 msgr2=0x562ff5a9ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- 172.21.15.70:0/2492751073 shutdown_connections 2024-04-15T15:15:17.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- 172.21.15.70:0/2492751073 wait complete. 2024-04-15T15:15:17.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 -- start start 2024-04-15T15:15:17.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.487+0000 7fe37ac06740 1 Processor -- start 2024-04-15T15:15:17.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5ad3f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5ad44b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562ff5aa9f00 con 0x562ff5a93e10 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe370200ad0 con 0x562ff5a941f0 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562ff5aab870 con 0x562ff5a9a120 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe37ac06740 10 client.0 _ll_register_callbacks cb 0x7fe360007040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54598/0 (socket says 172.21.15.70:54598) 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 -- 172.21.15.70:0/839146834 learned_addr learned my addr 172.21.15.70:0/839146834 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 -- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 msgr2=0x562ff5ad44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5ad44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 -- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 msgr2=0x562ff5ad3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5ad3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.488+0000 7fe375cdc640 1 -- 172.21.15.70:0/839146834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562ff5a96c70 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe375cdc640 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe36c07b2e0 tx=0x7fe36c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de87fccbe9b4900 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe36c031700 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe36c0779c0 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe36c085070 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fe36c09a050 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 10 client.4458 fetch_fsmap learned FSMap version 6 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x562ff5ac7ec0 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe36c085070 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7fe36c032070 con 0x562ff5a93e10 2024-04-15T15:15:17.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 10 client.4458 fetch_fsmap finished waiting for FSMap version 6 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 10 client.4458 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 20 client.4458 populate_metadata read hostname 'smithi070' 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5a839b0 0x562ff5b2f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 4 client.4458 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 --> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] -- command(tid 0: {"prefix": "status"}) -- 0x562ff5a68dc0 con 0x562ff5a839b0 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe3754db640 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5a839b0 0x562ff5b2f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe350ff9640 10 client.4458 ms_handle_connect on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.489+0000 7fe3754db640 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5a839b0 0x562ff5b2f050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d57a02a824a42806 server_cookie=eee7a3c397c62834 in_seq=0 out_seq=0 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe350ff9640 1 -- 172.21.15.70:0/839146834 <== mds.0 v2:172.21.15.70:6835/3048652815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x562ff5a68dc0 con 0x562ff5a839b0 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe350ff9640 10 client.4458 handle_command_reply: tid=0 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 client.4458 shutdown 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 msgr2=0x562ff5a98390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe36c07b2e0 tx=0x7fe36c02b7a0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 shutdown_connections 2024-04-15T15:15:17.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x562ff5a839b0 0x562ff5b2f050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562ff5a9a120 0x562ff5ad44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562ff5a941f0 0x562ff5ad3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 --2- 172.21.15.70:0/839146834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562ff5a93e10 0x562ff5a98390 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe350ff9640 0 client.4458 ms_handle_reset on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 >> 172.21.15.70:0/839146834 conn(0x562ff5a672f0 msgr2=0x562ff5aab9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 shutdown_connections 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 1 -- 172.21.15.70:0/839146834 wait complete. 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe37ac06740 20 client.4458 trim_cache size 0 max 16384 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:15:17.499 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "074678c8-a3ad-4fb8-89f1-2c7d457fbbe5", 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4293, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 7.2069573079999998, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 6, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 25, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 25, 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 13.865407861 2024-04-15T15:15:17.500 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 msgr2=0x7fe35406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 0x7fe35406edb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe3700b5ab0 tx=0x7fe360008040 comp rx=0 tx=0).stop 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 msgr2=0x7fe3701f38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe36c036040 tx=0x7fe36c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 shutdown_connections 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe37015e790 0x7fe3700cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3700d1b20 0x7fe3700cae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe35406c8f0 0x7fe35406edb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.490+0000 7fe377f40640 1 --2- 172.21.15.70:0/902923066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe3700b7be0 0x7fe3701f38c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.491+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 >> 172.21.15.70:0/902923066 conn(0x7fe370067b40 msgr2=0x7fe3700b9c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.491+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 shutdown_connections 2024-04-15T15:15:17.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.491+0000 7fe377f40640 1 -- 172.21.15.70:0/902923066 wait complete. 2024-04-15T15:15:17.504 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:17.504 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:17.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:17.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.603+0000 7f536efe1640 1 Processor -- start 2024-04-15T15:15:17.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.603+0000 7f536efe1640 1 -- start start 2024-04-15T15:15:17.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536efe1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f536806caf0 0x7f53680d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536efe1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f536806cff0 con 0x7f53680b1590 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536efe1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f536806d2d0 con 0x7f536809b0e0 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536efe1640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f536806dc00 con 0x7f536806caf0 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536cd7d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f53680b1590 0x7f536809a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50788/0 (socket says 172.21.15.70:50788) 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536cd7d640 1 -- 172.21.15.70:0/3393381631 learned_addr learned my addr 172.21.15.70:0/3393381631 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536d57e640 1 --2- 172.21.15.70:0/3393381631 >> v2:172.21.15.96:3300/0 conn(0x7f536806caf0 0x7f53680d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2837205143 0 0) 0x7f536806cff0 con 0x7f53680b1590 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5358003520 con 0x7f53680b1590 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2089036215 0 0) 0x7f5358003520 con 0x7f53680b1590 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 >> v2:172.21.15.96:3300/0 conn(0x7f536806caf0 msgr2=0x7f53680d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 --2- 172.21.15.70:0/3393381631 >> v2:172.21.15.96:3300/0 conn(0x7f536806caf0 0x7f53680d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 >> v1:172.21.15.96:6789/0 conn(0x7f536809b0e0 legacy=0x7f536815e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536806d5b0 con 0x7f53680b1590 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f535c001670 con 0x7f53680b1590 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f536d57e640 1 --2- 172.21.15.70:0/3393381631 >> v2:172.21.15.96:3300/0 conn(0x7f536806caf0 0x7f53680d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 >> v1:172.21.15.70:6789/0 conn(0x7f53680b1590 legacy=0x7f536809a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f53580039b0 0x7f5358003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 0x7f5358004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53580051c0 0x7f535800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f536806dc00 con 0x7f5358004800 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f535800c110 con 0x7f53580051c0 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.604+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f535800c310 con 0x7f53580039b0 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 0x7f5358004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f536cd7d640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f53580039b0 0x7f5358003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f536d57e640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53580051c0 0x7f535800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 -- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f53580039b0 msgr2=0x7f5358003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f53580039b0 0x7f5358003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 -- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53580051c0 msgr2=0x7f535800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53580051c0 0x7f535800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f5367fff640 1 -- 172.21.15.70:0/3393381631 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5358003520 con 0x7f5358004800 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.605+0000 7f536d57e640 1 -- 172.21.15.70:0/3393381631 reap_dead start 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f5367fff640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 0x7f5358004c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5354001750 tx=0x7f5358005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3377b08e9ff2acc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354001370 con 0x7f5358004800 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53540052d0 con 0x7f5358004800 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f53677fe640 1 -- 172.21.15.70:0/3393381631 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354003830 con 0x7f5358004800 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 -- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 msgr2=0x7f5358004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 0x7f5358004c80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5354001750 tx=0x7f5358005830 comp rx=0 tx=0).stop 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 -- 172.21.15.70:0/3393381631 shutdown_connections 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 --2- 172.21.15.70:0/3393381631 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5358004800 0x7f5358004c80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 -- 172.21.15.70:0/3393381631 >> 172.21.15.70:0/3393381631 conn(0x7f53680b3e90 msgr2=0x7f53680b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 -- 172.21.15.70:0/3393381631 shutdown_connections 2024-04-15T15:15:17.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.606+0000 7f536efe1640 1 -- 172.21.15.70:0/3393381631 wait complete. 2024-04-15T15:15:17.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.607+0000 7f536efe1640 1 Processor -- start 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 -- start start 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5358004800 0x7f5368156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53680b7190 0x7f53680b7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f53680d0b20 con 0x7f53680b5310 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f53680d09a0 con 0x7f53680b7190 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f536efe1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f53680b1970 con 0x7f5358004800 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f5367fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.608+0000 7f5367fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54624/0 (socket says 172.21.15.70:54624) 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 -- 172.21.15.70:0/2386394878 learned_addr learned my addr 172.21.15.70:0/2386394878 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f536d57e640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53680b7190 0x7f53680b7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f536cd7d640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5358004800 0x7f5368156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 -- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5358004800 msgr2=0x7f5368156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5358004800 0x7f5368156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 -- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53680b7190 msgr2=0x7f53680b7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53680b7190 0x7f53680b7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 -- 172.21.15.70:0/2386394878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53680b58b0 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f5367fff640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5354000c00 tx=0x7f5354001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354041020 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5354005480 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53680b5b90 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354007a30 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.609+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53680b6150 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.610+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53680689e0 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.611+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5354044020 con 0x7f53680b5310 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.611+0000 7f53657fa640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:17.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.611+0000 7f536cd7d640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:17.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.611+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f53540ad7f0 con 0x7f53680b5310 2024-04-15T15:15:17.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.612+0000 7f536cd7d640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 0x7f534c06ede0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f535c001050 tx=0x7f535c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:17.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.616+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f535404b030 con 0x7f53680b5310 2024-04-15T15:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.828+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f53680b4a80 con 0x7f53680b5310 2024-04-15T15:15:17.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.828+0000 7f53657fa640 1 -- 172.21.15.70:0/2386394878 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7f53540303c0 con 0x7f53680b5310 2024-04-15T15:15:17.831 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 msgr2=0x7f534c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 0x7f534c06ede0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f535c001050 tx=0x7f535c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 msgr2=0x7f536815cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:17.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5354000c00 tx=0x7f5354001120 comp rx=0 tx=0).stop 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 shutdown_connections 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f53680b7190 0x7f53680b7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f534c06c920 0x7f534c06ede0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f53680b5310 0x7f536815cef0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 --2- 172.21.15.70:0/2386394878 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5358004800 0x7f5368156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 >> 172.21.15.70:0/2386394878 conn(0x7f53680b3e90 msgr2=0x7f53680b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 shutdown_connections 2024-04-15T15:15:17.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17.830+0000 7f536efe1640 1 -- 172.21.15.70:0/2386394878 wait complete. 2024-04-15T15:15:17.844 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-15T15:15:17.844 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:17.844 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:17.845 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:15:17.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:17.845 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:17.861 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:17.862 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:17.862 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:17.862 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:17.863 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:17.864 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:17.864 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:17.864 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42733sec preferred_lft 42733sec 2024-04-15T15:15:17.864 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:15:17.864 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:17.864 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:17.865 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:17.865 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:17.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:17.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:17.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-15T15:15:17.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:17.866 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:17.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:18.076 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:18.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:18.082 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:18.111 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:15:18.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:18.113 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:15:18.196 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:15:18.196 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:18.196 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:18.196 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:18.196 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:18.197 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:18.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:18.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:18.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:18.487 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:15:18.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:18.504 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-15T15:15:18.559 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:18.559 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:18.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:18.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-15T15:15:18.559 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:18.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:18.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:18.684 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-15T15:15:18.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-15T15:15:18.685 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-15T15:15:18.685 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:18.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:18.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:18.822 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:18.822 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:18.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-15T15:15:18.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:18.823 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:18.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:18.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:18.906 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-15T15:15:18.906 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:18.906 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:18.934 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-15T15:15:18.934 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:18.934 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:18.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:19.059 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:19.115 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:19.115 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:19.115 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:19.116 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:19.117 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:19.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.118 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:19.171 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:19.171 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-15T15:15:19.216 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:19.244 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:19.302 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:19.303 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:19.304 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:19.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.306 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:19.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:2024-04-15T15:15:19.306+0000 7fcc47845480 -1 init, newargv = 0x556fc1cc2b50 newargc=15 2024-04-15T15:15:19.307 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[43796]: starting ceph client 2024-04-15T15:15:19.313 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[43796]: starting fuse 2024-04-15T15:15:19.322 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:15:19.323 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-15T15:15:19.323 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:19.324 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.admin")) 2024-04-15T15:15:19.325 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:15:19.445 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.admin.43796.asok 2024-04-15T15:15:19.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:19.455 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43796.asok 2024-04-15T15:15:19.455 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43796.asok status 2024-04-15T15:15:19.593 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:15:19.593 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:15:19.593 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:19.593 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "admin", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "43796", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4470, 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4470 2024-04-15T15:15:19.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2151581546 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2151581546 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4470 192.168.0.1:0/2151581546", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/2151581546", 2024-04-15T15:15:19.595 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 6, 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 25, 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:15:19.596 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:15:19.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.602 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-15T15:15:19.602 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:19.618 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-15T15:15:19.618 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:19.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:19.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:19.688 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-15T15:15:19.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:15:19.792 INFO:teuthology.orchestra.run.smithi070.stdout:# file: . 2024-04-15T15:15:19.792 INFO:teuthology.orchestra.run.smithi070.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-15T15:15:19.792 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:19.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:19.798 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:15:19.798 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:19.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.955+0000 7f440d850640 1 Processor -- start 2024-04-15T15:15:19.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.955+0000 7f440d850640 1 -- start start 2024-04-15T15:15:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f440d850640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 0x7f44080ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f440d850640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f44080b0560 0x7f440806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f440d850640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44080cbd10 con 0x7f44080b09a0 2024-04-15T15:15:19.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f440d850640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f440815eee0 con 0x7f44080b0560 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f440d850640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f44080cbff0 con 0x7f440815eb00 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f44067fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f44080b0560 0x7f440806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f44067fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f44080b0560 0x7f440806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54630/0 (socket says 172.21.15.70:54630) 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.956+0000 7f44067fc640 1 -- 172.21.15.70:0/151514597 learned_addr learned my addr 172.21.15.70:0/151514597 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f4406ffd640 1 --2- 172.21.15.70:0/151514597 >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 0x7f44080ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f44067fc640 1 -- 172.21.15.70:0/151514597 >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 msgr2=0x7f44080ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f44067fc640 1 --2- 172.21.15.70:0/151514597 >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 0x7f44080ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f44067fc640 1 -- 172.21.15.70:0/151514597 >> v1:172.21.15.96:6789/0 conn(0x7f44080b09a0 legacy=0x7f44080ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f44067fc640 1 -- 172.21.15.70:0/151514597 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44080cc2d0 con 0x7f44080b0560 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f4406ffd640 1 --2- 172.21.15.70:0/151514597 >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 0x7f44080ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.957+0000 7f44067fc640 1 --2- 172.21.15.70:0/151514597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44080b0560 0x7f440806b9e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4408073cd0 tx=0x7f43f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d22caa319c9035 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f4405ffb640 1 -- 172.21.15.70:0/151514597 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0034070 con 0x7f44080b0560 2024-04-15T15:15:19.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f4405ffb640 1 -- 172.21.15.70:0/151514597 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43f0030860 con 0x7f44080b0560 2024-04-15T15:15:19.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f4405ffb640 1 -- 172.21.15.70:0/151514597 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43f0030c30 con 0x7f44080b0560 2024-04-15T15:15:19.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 -- 172.21.15.70:0/151514597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44080b0560 msgr2=0x7f440806b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 --2- 172.21.15.70:0/151514597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44080b0560 0x7f440806b9e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4408073cd0 tx=0x7f43f002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 -- 172.21.15.70:0/151514597 shutdown_connections 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 --2- 172.21.15.70:0/151514597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44080b0560 0x7f440806b9e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 --2- 172.21.15.70:0/151514597 >> v2:172.21.15.96:3300/0 conn(0x7f440815eb00 0x7f44080ae2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 -- 172.21.15.70:0/151514597 >> 172.21.15.70:0/151514597 conn(0x7f440809a9d0 msgr2=0x7f44080d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 -- 172.21.15.70:0/151514597 shutdown_connections 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.958+0000 7f440d850640 1 -- 172.21.15.70:0/151514597 wait complete. 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.959+0000 7f440d850640 1 Processor -- start 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 -- start start 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44080b0560 0x7f44080c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 0x7f4408155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44080ae7e0 con 0x7f440815eb00 2024-04-15T15:15:19.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f44080cc890 con 0x7f44080b09a0 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f440d850640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f44080cec60 con 0x7f44080b0560 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f44077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f44077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54646/0 (socket says 172.21.15.70:54646) 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.960+0000 7f44077fe640 1 -- 172.21.15.70:0/2373847770 learned_addr learned my addr 172.21.15.70:0/2373847770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44067fc640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 0x7f4408155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f4406ffd640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44080b0560 0x7f44080c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 -- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44080b0560 msgr2=0x7f44080c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44080b0560 0x7f44080c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 -- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 msgr2=0x7f4408155da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 0x7f4408155da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 -- 172.21.15.70:0/2373847770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44080c6420 con 0x7f440815eb00 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44067fc640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 0x7f4408155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f44077fe640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4400001eb0 tx=0x7f440000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400017070 con 0x7f440815eb00 2024-04-15T15:15:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44000052d0 con 0x7f440815eb00 2024-04-15T15:15:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44080c6700 con 0x7f440815eb00 2024-04-15T15:15:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4400003810 con 0x7f440815eb00 2024-04-15T15:15:19.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.961+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44080c6cc0 con 0x7f440815eb00 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.962+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44080690f0 con 0x7f440815eb00 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.963+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f440001a070 con 0x7f440815eb00 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.963+0000 7f43fffff640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.963+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f440008b230 con 0x7f440815eb00 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.963+0000 7f4406ffd640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:19.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.964+0000 7f4406ffd640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 0x7f43e806edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009ed0 tx=0x7f43f800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:19.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:19.968+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4400059320 con 0x7f440815eb00 2024-04-15T15:15:20.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.147+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f44080ce6a0 con 0x7f440815eb00 2024-04-15T15:15:20.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.148+0000 7f43fffff640 1 -- 172.21.15.70:0/2373847770 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f4400058c20 con 0x7f440815eb00 2024-04-15T15:15:20.150 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:20.151 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 msgr2=0x7f43e806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 0x7f43e806edb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009ed0 tx=0x7f43f800b040 comp rx=0 tx=0).stop 2024-04-15T15:15:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 msgr2=0x7f44080cb060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4400001eb0 tx=0x7f440000c890 comp rx=0 tx=0).stop 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 shutdown_connections 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f43e806c8f0 0x7f43e806edb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f440815eb00 0x7f44080cb060 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44080b09a0 0x7f4408155da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 --2- 172.21.15.70:0/2373847770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44080b0560 0x7f44080c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.150+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 >> 172.21.15.70:0/2373847770 conn(0x7f440809a9d0 msgr2=0x7f440806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.151+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 shutdown_connections 2024-04-15T15:15:20.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.151+0000 7f440d850640 1 -- 172.21.15.70:0/2373847770 wait complete. 2024-04-15T15:15:20.164 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:20.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:20.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.267+0000 7f8bade55640 1 Processor -- start 2024-04-15T15:15:20.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.267+0000 7f8bade55640 1 -- start start 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8bade55640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8ba80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8bade55640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba815eb70 con 0x7f8ba815e790 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8bade55640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba80d3c80 con 0x7f8ba80d4a40 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8bade55640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba80d3790 con 0x7f8ba80d15e0 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8ba77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8ba815e790 0x7f8ba80b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50800/0 (socket says 172.21.15.70:50800) 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8ba77fe640 1 -- 172.21.15.70:0/1882851288 learned_addr learned my addr 172.21.15.70:0/1882851288 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2506194033 0 0) 0x7f8ba815eb70 con 0x7f8ba815e790 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.268+0000 7f8ba6ffd640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8ba80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b90003520 con 0x7f8ba815e790 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 917015621 0 0) 0x7f8b90003520 con 0x7f8ba815e790 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 msgr2=0x7f8ba80d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8ba80d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 >> v1:172.21.15.96:6789/0 conn(0x7f8ba80d4a40 legacy=0x7f8ba80d29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba80d3f60 con 0x7f8ba815e790 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f8b98001a10 con 0x7f8ba815e790 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 >> v1:172.21.15.70:6789/0 conn(0x7f8ba815e790 legacy=0x7f8ba80b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900039e0 0x7f8b90003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8b90004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b90004d40 0x7f8b900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80d3790 con 0x7f8b900039e0 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b9000bb80 con 0x7f8b90004d40 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8b9000bd80 con 0x7f8ba80d15e0 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900039e0 0x7f8b90003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba7fff640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b90004d40 0x7f8b900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba6ffd640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8b90004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 -- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 msgr2=0x7f8b90004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8b90004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 -- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b90004d40 msgr2=0x7f8b900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b90004d40 0x7f8b900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba77fe640 1 -- 172.21.15.70:0/1882851288 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b90003520 con 0x7f8b900039e0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.269+0000 7f8ba6ffd640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ba80d15e0 0x7f8b90004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.270+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900039e0 0x7f8b90003e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800a830 tx=0x7f8b90005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b0aeeb3898e1c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.270+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9800a480 con 0x7f8b900039e0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.270+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b98034630 con 0x7f8b900039e0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.270+0000 7f8ba67fc640 1 -- 172.21.15.70:0/1882851288 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98034a20 con 0x7f8b900039e0 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 -- 172.21.15.70:0/1882851288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900039e0 msgr2=0x7f8b90003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 --2- 172.21.15.70:0/1882851288 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b900039e0 0x7f8b90003e00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8b9800a830 tx=0x7f8b90005830 comp rx=0 tx=0).stop 2024-04-15T15:15:20.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8ba7fff640 1 -- 172.21.15.70:0/1882851288 reap_dead start 2024-04-15T15:15:20.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 -- 172.21.15.70:0/1882851288 shutdown_connections 2024-04-15T15:15:20.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 -- 172.21.15.70:0/1882851288 >> 172.21.15.70:0/1882851288 conn(0x7f8ba8067b40 msgr2=0x7f8ba8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 -- 172.21.15.70:0/1882851288 shutdown_connections 2024-04-15T15:15:20.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.271+0000 7f8bade55640 1 -- 172.21.15.70:0/1882851288 wait complete. 2024-04-15T15:15:20.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.272+0000 7f8bade55640 1 Processor -- start 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.272+0000 7f8bade55640 1 -- start start 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b900039e0 0x7f8ba815a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 0x7f8ba8158c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80b7db0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80d45b0 con 0x7f8ba81587e0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba80d30d0 con 0x7f8b900039e0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54666/0 (socket says 172.21.15.70:54666) 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 -- 172.21.15.70:0/4228612589 learned_addr learned my addr 172.21.15.70:0/4228612589 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b900039e0 0x7f8ba815a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba6ffd640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 0x7f8ba8158c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 -- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b900039e0 msgr2=0x7f8ba815a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b900039e0 0x7f8ba815a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 -- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 msgr2=0x7f8ba8158c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 0x7f8ba8158c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 -- 172.21.15.70:0/4228612589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba8159d70 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba6ffd640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 0x7f8ba8158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8ba7fff640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80cec10 tx=0x7f8ba000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba00053a0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ba0018070 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba81594d0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.273+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ba0003830 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.274+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba8159a90 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.274+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba80689e0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.275+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ba00039d0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.275+0000 7f8b8ffff640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 0x7f8b9406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.275+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8ba008b860 con 0x7f8ba81f76e0 2024-04-15T15:15:20.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.275+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 0x7f8b9406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.276+0000 7f8ba77fe640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 0x7f8b9406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b98034000 tx=0x7f8b98009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.279+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ba00599e0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.456+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8ba80d0520 con 0x7f8ba81f76e0 2024-04-15T15:15:20.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.456+0000 7f8b8ffff640 1 -- 172.21.15.70:0/4228612589 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f8ba00592e0 con 0x7f8ba81f76e0 2024-04-15T15:15:20.458 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:20.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 msgr2=0x7f8b9406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 0x7f8b9406edb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b98034000 tx=0x7f8b98009030 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 msgr2=0x7f8ba81f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8ba80cec10 tx=0x7f8ba000f890 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 shutdown_connections 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b9406c8f0 0x7f8b9406edb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ba81f76e0 0x7f8ba81f7b60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ba81587e0 0x7f8ba8158c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 --2- 172.21.15.70:0/4228612589 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b900039e0 0x7f8ba815a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 >> 172.21.15.70:0/4228612589 conn(0x7f8ba8067b40 msgr2=0x7f8ba80cde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 shutdown_connections 2024-04-15T15:15:20.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.458+0000 7f8bade55640 1 -- 172.21.15.70:0/4228612589 wait complete. 2024-04-15T15:15:20.471 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:20.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.576+0000 7f06094ce640 1 Processor -- start 2024-04-15T15:15:20.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.576+0000 7f06094ce640 1 -- start start 2024-04-15T15:15:20.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f060415e700 0x7f06040b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f06040d2360 0x7f06040d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f06040d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f06040b53b0 con 0x7f060415e700 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f06040b47a0 con 0x7f06040d2360 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06094ce640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f06040b4a80 con 0x7f060415eae0 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f060415e700 0x7f06040b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f060415e700 0x7f06040b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54682/0 (socket says 172.21.15.70:54682) 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 -- 172.21.15.70:0/779953240 learned_addr learned my addr 172.21.15.70:0/779953240 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f06037fe640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f06040d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f05fa7fc640 1 --2- 172.21.15.70:0/779953240 >> v2:172.21.15.96:3300/0 conn(0x7f06040d2360 0x7f06040d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 -- 172.21.15.70:0/779953240 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 msgr2=0x7f06040d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f06040d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 -- 172.21.15.70:0/779953240 >> v2:172.21.15.96:3300/0 conn(0x7f06040d2360 msgr2=0x7f06040d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 --2- 172.21.15.70:0/779953240 >> v2:172.21.15.96:3300/0 conn(0x7f06040d2360 0x7f06040d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.577+0000 7f0602ffd640 1 -- 172.21.15.70:0/779953240 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06040b4d60 con 0x7f060415e700 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.578+0000 7f0602ffd640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f060415e700 0x7f06040b5840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f060408c7b0 tx=0x7f05f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea3700b75947dded server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.578+0000 7f06027fc640 1 -- 172.21.15.70:0/779953240 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f402d040 con 0x7f060415e700 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.578+0000 7f06027fc640 1 -- 172.21.15.70:0/779953240 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f4004230 con 0x7f060415e700 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.578+0000 7f06027fc640 1 -- 172.21.15.70:0/779953240 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4003610 con 0x7f060415e700 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 -- 172.21.15.70:0/779953240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f060415e700 msgr2=0x7f06040b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f060415e700 0x7f06040b5840 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f060408c7b0 tx=0x7f05f402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 -- 172.21.15.70:0/779953240 shutdown_connections 2024-04-15T15:15:20.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f06040d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 --2- 172.21.15.70:0/779953240 >> v2:172.21.15.96:3300/0 conn(0x7f06040d2360 0x7f06040d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 --2- 172.21.15.70:0/779953240 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f060415e700 0x7f06040b5840 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 -- 172.21.15.70:0/779953240 >> 172.21.15.70:0/779953240 conn(0x7f060408bf60 msgr2=0x7f06040d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 -- 172.21.15.70:0/779953240 shutdown_connections 2024-04-15T15:15:20.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.579+0000 7f06094ce640 1 -- 172.21.15.70:0/779953240 wait complete. 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.580+0000 7f06094ce640 1 Processor -- start 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.580+0000 7f06094ce640 1 -- start start 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f060415e700 0x7f06040cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f0604152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f06040b5d80 con 0x7f06040d2360 2024-04-15T15:15:20.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f06040b4450 con 0x7f060415e700 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f06040d1d40 con 0x7f060415eae0 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54696/0 (socket says 172.21.15.70:54696) 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 -- 172.21.15.70:0/2134517232 learned_addr learned my addr 172.21.15.70:0/2134517232 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06037fe640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f0604152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f05fa7fc640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f060415e700 0x7f06040cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 -- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 msgr2=0x7f0604152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f0604152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 -- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f060415e700 msgr2=0x7f06040cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f060415e700 0x7f06040cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 -- 172.21.15.70:0/2134517232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06040b8340 con 0x7f06040d2360 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f0602ffd640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05f40090b0 tx=0x7f05f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4003c50 con 0x7f06040d2360 2024-04-15T15:15:20.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05f4003e10 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05f4006280 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.581+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06040b8620 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.582+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06040b8be0 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.582+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06040689e0 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.583+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f05f4006420 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.583+0000 7f05fbfff640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 0x7f05e406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.583+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f05f40ae080 con 0x7f06040d2360 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.584+0000 7f05fa7fc640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 0x7f05e406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.590 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.584+0000 7f05fa7fc640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 0x7f05e406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0604159c10 tx=0x7f05ec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.588+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05f407c170 con 0x7f06040d2360 2024-04-15T15:15:20.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.764+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f06040ba500 con 0x7f06040d2360 2024-04-15T15:15:20.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.764+0000 7f05fbfff640 1 -- 172.21.15.70:0/2134517232 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f05f407ba70 con 0x7f06040d2360 2024-04-15T15:15:20.766 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:20.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 msgr2=0x7f05e406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 0x7f05e406edb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0604159c10 tx=0x7f05ec008040 comp rx=0 tx=0).stop 2024-04-15T15:15:20.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 msgr2=0x7f06040b93e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f05f40090b0 tx=0x7f05f40046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 shutdown_connections 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f060415eae0 0x7f0604152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f060415e700 0x7f06040cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f05e406c8f0 0x7f05e406edb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 --2- 172.21.15.70:0/2134517232 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06040d2360 0x7f06040b93e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 >> 172.21.15.70:0/2134517232 conn(0x7f060408bf60 msgr2=0x7f060415f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 shutdown_connections 2024-04-15T15:15:20.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.766+0000 7f06094ce640 1 -- 172.21.15.70:0/2134517232 wait complete. 2024-04-15T15:15:20.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:20.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 Processor -- start 2024-04-15T15:15:20.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 -- start start 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe84c0b5230 0x7fe84c072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe84c15c1d0 con 0x7fe84c072c60 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe84c0b4da0 con 0x7fe84c0b5230 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.883+0000 7fe852e50640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c0b4190 con 0x7fe84c158770 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe850bec640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe84c0b5230 0x7fe84c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe850bec640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43484/0 (socket says 172.21.15.70:43484) 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe850bec640 1 -- 172.21.15.70:0/2358915221 learned_addr learned my addr 172.21.15.70:0/2358915221 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:20.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe84c0b5230 0x7fe84c072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:33828/0 (socket says 172.21.15.70:33828) 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 -- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 msgr2=0x7fe84c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 -- 172.21.15.70:0/2358915221 >> v1:172.21.15.96:6789/0 conn(0x7fe84c072c60 legacy=0x7fe84c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84bfff640 1 -- 172.21.15.70:0/2358915221 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c0b4470 con 0x7fe84c0b5230 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe84b7fe640 1 -- 172.21.15.70:0/2358915221 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121442932 0 0) 0x7fe844001490 con 0x7fe84c072c60 2024-04-15T15:15:20.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.884+0000 7fe850bec640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.885+0000 7fe84bfff640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c0b5230 0x7fe84c072720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe83800a940 tx=0x7fe83802cd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f66e7295c0ac903 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.885+0000 7fe84b7fe640 1 -- 172.21.15.70:0/2358915221 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83803a070 con 0x7fe84c0b5230 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.885+0000 7fe84b7fe640 1 -- 172.21.15.70:0/2358915221 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83802db30 con 0x7fe84c0b5230 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.885+0000 7fe84b7fe640 1 -- 172.21.15.70:0/2358915221 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83800b040 con 0x7fe84c0b5230 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 -- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c0b5230 msgr2=0x7fe84c072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c0b5230 0x7fe84c072720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe83800a940 tx=0x7fe83802cd80 comp rx=0 tx=0).stop 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 -- 172.21.15.70:0/2358915221 shutdown_connections 2024-04-15T15:15:20.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c0b5230 0x7fe84c072720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 --2- 172.21.15.70:0/2358915221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c158770 0x7fe84c15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 -- 172.21.15.70:0/2358915221 >> 172.21.15.70:0/2358915221 conn(0x7fe84c06be00 msgr2=0x7fe84c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 -- 172.21.15.70:0/2358915221 shutdown_connections 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.886+0000 7fe852e50640 1 -- 172.21.15.70:0/2358915221 wait complete. 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.887+0000 7fe852e50640 1 Processor -- start 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.887+0000 7fe852e50640 1 -- start start 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c0b5230 0x7fe84c159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c158770 0x7fe84c159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c071fa0 con 0x7fe84c072c60 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe84c0b4a30 con 0x7fe84c158770 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe852e50640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe84c158b50 con 0x7fe84c0b5230 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54706/0 (socket says 172.21.15.70:54706) 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 -- 172.21.15.70:0/967694092 learned_addr learned my addr 172.21.15.70:0/967694092 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe8513ed640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c158770 0x7fe84c159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe84bfff640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c0b5230 0x7fe84c159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 -- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c0b5230 msgr2=0x7fe84c159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c0b5230 0x7fe84c159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 -- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c158770 msgr2=0x7fe84c159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c158770 0x7fe84c159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 -- 172.21.15.70:0/967694092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84c1e48d0 con 0x7fe84c072c60 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.888+0000 7fe850bec640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe84000e9b0 tx=0x7fe84000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.889+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe840017070 con 0x7fe84c072c60 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.889+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84000c9a0 con 0x7fe84c072c60 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.889+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84000cd50 con 0x7fe84c072c60 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.889+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84c154fa0 con 0x7fe84c072c60 2024-04-15T15:15:20.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.889+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe84c155560 con 0x7fe84c072c60 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.890+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84c0689e0 con 0x7fe84c072c60 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.890+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe84001a070 con 0x7fe84c072c60 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.890+0000 7fe8497fa640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 0x7fe83006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.890+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe84008bbe0 con 0x7fe84c072c60 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.891+0000 7fe84bfff640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 0x7fe83006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:20.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.891+0000 7fe84bfff640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 0x7fe83006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe838037000 tx=0x7fe838031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:20.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:20.895+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe840059710 con 0x7fe84c072c60 2024-04-15T15:15:21.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.070+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe84c0afb90 con 0x7fe84c072c60 2024-04-15T15:15:21.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.070+0000 7fe8497fa640 1 -- 172.21.15.70:0/967694092 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe84005da10 con 0x7fe84c072c60 2024-04-15T15:15:21.072 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:21.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 msgr2=0x7fe83006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.074 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 0x7fe83006edb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe838037000 tx=0x7fe838031000 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 msgr2=0x7fe84c155d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe84000e9b0 tx=0x7fe84000ee70 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 shutdown_connections 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe84c158770 0x7fe84c159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe84c0b5230 0x7fe84c159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe83006c8f0 0x7fe83006edb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 --2- 172.21.15.70:0/967694092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe84c072c60 0x7fe84c155d60 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 >> 172.21.15.70:0/967694092 conn(0x7fe84c06be00 msgr2=0x7fe84c0b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 shutdown_connections 2024-04-15T15:15:21.075 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.072+0000 7fe852e50640 1 -- 172.21.15.70:0/967694092 wait complete. 2024-04-15T15:15:21.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:21.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.189+0000 7f4375b7c640 1 Processor -- start 2024-04-15T15:15:21.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.189+0000 7f4375b7c640 1 -- start start 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f4375b7c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f4375b7c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 0x7f437015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f4375b7c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4370070720 con 0x7f437009adb0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f4375b7c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f437009ba90 con 0x7f43700701f0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f4375b7c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4370160070 con 0x7f437009a9d0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f437009adb0 0x7f43700d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50810/0 (socket says 172.21.15.70:50810) 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436ffff640 1 -- 172.21.15.70:0/1903272630 learned_addr learned my addr 172.21.15.70:0/1903272630 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436f7fe640 1 --2- 172.21.15.70:0/1903272630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436effd640 1 --2- 172.21.15.70:0/1903272630 >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 0x7f437015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4011187441 0 0) 0x7f4370070720 con 0x7f437009adb0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.190+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4358003520 con 0x7f437009adb0 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436f7fe640 1 -- 172.21.15.70:0/1903272630 >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 msgr2=0x7f437015fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:21.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436f7fe640 1 --2- 172.21.15.70:0/1903272630 >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 0x7f437015fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1036988937 0 0) 0x7f4358003520 con 0x7f437009adb0 2024-04-15T15:15:21.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436f7fe640 1 -- 172.21.15.70:0/1903272630 >> v1:172.21.15.70:6789/0 conn(0x7f437009adb0 legacy=0x7f43700d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436f7fe640 1 -- 172.21.15.70:0/1903272630 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4370070a90 con 0x7f437009a9d0 2024-04-15T15:15:21.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436effd640 1 --2- 172.21.15.70:0/1903272630 >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 0x7f437015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:21.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.191+0000 7f436f7fe640 1 --2- 172.21.15.70:0/1903272630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370160420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43600097d0 tx=0x7f436002b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f66a59d0a6b7b850 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360037070 con 0x7f437009a9d0 2024-04-15T15:15:21.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4360030510 con 0x7f437009a9d0 2024-04-15T15:15:21.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f436e7fc640 1 -- 172.21.15.70:0/1903272630 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43680019f0 con 0x7f437009a9d0 2024-04-15T15:15:21.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 -- 172.21.15.70:0/1903272630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 msgr2=0x7f4370160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 --2- 172.21.15.70:0/1903272630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370160420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43600097d0 tx=0x7f436002b770 comp rx=0 tx=0).stop 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 -- 172.21.15.70:0/1903272630 shutdown_connections 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 --2- 172.21.15.70:0/1903272630 >> v2:172.21.15.96:3300/0 conn(0x7f43700701f0 0x7f437015fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 --2- 172.21.15.70:0/1903272630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370160420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 -- 172.21.15.70:0/1903272630 >> 172.21.15.70:0/1903272630 conn(0x7f43700d2460 msgr2=0x7f43700d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 -- 172.21.15.70:0/1903272630 shutdown_connections 2024-04-15T15:15:21.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.192+0000 7f4375b7c640 1 -- 172.21.15.70:0/1903272630 wait complete. 2024-04-15T15:15:21.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.193+0000 7f4375b7c640 1 Processor -- start 2024-04-15T15:15:21.197 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.193+0000 7f4375b7c640 1 -- start start 2024-04-15T15:15:21.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 0x7f437006db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4370071050 con 0x7f43700701f0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f43700d08c0 con 0x7f437009adb0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f4375b7c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f43700b1c60 con 0x7f437009a9d0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f436f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f436f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54716/0 (socket says 172.21.15.70:54716) 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f436f7fe640 1 -- 172.21.15.70:0/3327144300 learned_addr learned my addr 172.21.15.70:0/3327144300 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f436ffff640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 0x7f437006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.194+0000 7f436effd640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 -- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 msgr2=0x7f4370150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 -- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 msgr2=0x7f437006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 0x7f437006db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 -- 172.21.15.70:0/3327144300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43700af6e0 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436ffff640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 0x7f437006db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f436f7fe640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f436002b7b0 tx=0x7f43600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360042070 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f436003e870 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4360037040 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43700af9c0 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.195+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43700aff80 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.196+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43700689e0 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.196+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4360045070 con 0x7f43700701f0 2024-04-15T15:15:21.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.197+0000 7f4357fff640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 0x7f435c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.197+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f43600adab0 con 0x7f43700701f0 2024-04-15T15:15:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.197+0000 7f436effd640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 0x7f435c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.197+0000 7f436effd640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 0x7f435c06edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43700cca10 tx=0x7f4364008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.201+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f436007bb20 con 0x7f43700701f0 2024-04-15T15:15:21.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.376+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f43700d3370 con 0x7f43700701f0 2024-04-15T15:15:21.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.376+0000 7f4357fff640 1 -- 172.21.15.70:0/3327144300 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f436007b420 con 0x7f43700701f0 2024-04-15T15:15:21.378 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.378+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 msgr2=0x7f435c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.378+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 0x7f435c06edb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f43700cca10 tx=0x7f4364008040 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.378+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 msgr2=0x7f43700b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.378+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f436002b7b0 tx=0x7f43600046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 shutdown_connections 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437009adb0 0x7f437006db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437009a9d0 0x7f4370150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06c8f0 0x7f435c06edb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 --2- 172.21.15.70:0/3327144300 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f43700701f0 0x7f43700b0eb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 >> 172.21.15.70:0/3327144300 conn(0x7f43700d2460 msgr2=0x7f4370160460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 shutdown_connections 2024-04-15T15:15:21.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.379+0000 7f4375b7c640 1 -- 172.21.15.70:0/3327144300 wait complete. 2024-04-15T15:15:21.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:21.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.493+0000 7fd7da509640 1 Processor -- start 2024-04-15T15:15:21.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.493+0000 7fd7da509640 1 -- start start 2024-04-15T15:15:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7da509640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7d40d0da0 0x7fd7d406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7da509640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d415db40 con 0x7fd7d4071d10 2024-04-15T15:15:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7da509640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7d4072170 con 0x7fd7d4072d80 2024-04-15T15:15:21.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7da509640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd7d415cfc0 con 0x7fd7d40d0da0 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d8aa6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd7d40d0da0 0x7fd7d406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd7d4071d10 0x7fd7d40d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50820/0 (socket says 172.21.15.70:50820) 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d37fe640 1 -- 172.21.15.70:0/1396366197 learned_addr learned my addr 172.21.15.70:0/1396366197 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d8aa6640 1 -- 172.21.15.70:0/1396366197 >> v1:172.21.15.96:6789/0 conn(0x7fd7d4072d80 legacy=0x7fd7d40af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d8aa6640 1 -- 172.21.15.70:0/1396366197 >> v1:172.21.15.70:6789/0 conn(0x7fd7d4071d10 legacy=0x7fd7d40d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d2ffd640 1 -- 172.21.15.70:0/1396366197 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 355518172 0 0) 0x7fd7d415db40 con 0x7fd7d4071d10 2024-04-15T15:15:21.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.494+0000 7fd7d8aa6640 1 -- 172.21.15.70:0/1396366197 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d415d2a0 con 0x7fd7d40d0da0 2024-04-15T15:15:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.495+0000 7fd7d8aa6640 1 --2- 172.21.15.70:0/1396366197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d40d0da0 0x7fd7d406e060 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4072340 tx=0x7fd7cc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92494bd2caff4597 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.495+0000 7fd7d2ffd640 1 -- 172.21.15.70:0/1396366197 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc039070 con 0x7fd7d40d0da0 2024-04-15T15:15:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.495+0000 7fd7d2ffd640 1 -- 172.21.15.70:0/1396366197 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7cc02cc50 con 0x7fd7d40d0da0 2024-04-15T15:15:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.495+0000 7fd7d2ffd640 1 -- 172.21.15.70:0/1396366197 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7cc034440 con 0x7fd7d40d0da0 2024-04-15T15:15:21.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 -- 172.21.15.70:0/1396366197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d40d0da0 msgr2=0x7fd7d406e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 --2- 172.21.15.70:0/1396366197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d40d0da0 0x7fd7d406e060 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4072340 tx=0x7fd7cc02bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 -- 172.21.15.70:0/1396366197 shutdown_connections 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 --2- 172.21.15.70:0/1396366197 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d40d0da0 0x7fd7d406e060 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 -- 172.21.15.70:0/1396366197 >> 172.21.15.70:0/1396366197 conn(0x7fd7d406be00 msgr2=0x7fd7d406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 -- 172.21.15.70:0/1396366197 shutdown_connections 2024-04-15T15:15:21.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.496+0000 7fd7da509640 1 -- 172.21.15.70:0/1396366197 wait complete. 2024-04-15T15:15:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.497+0000 7fd7da509640 1 Processor -- start 2024-04-15T15:15:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.497+0000 7fd7da509640 1 -- start start 2024-04-15T15:15:21.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 0x7fd7d4158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 0x7fd7d4153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d40afaf0 con 0x7fd7d4072d80 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd7d415d860 con 0x7fd7d4071d10 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7da509640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd7d415cb90 con 0x7fd7d40d0da0 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d37fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54736/0 (socket says 172.21.15.70:54736) 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d37fe640 1 -- 172.21.15.70:0/2417835480 learned_addr learned my addr 172.21.15.70:0/2417835480 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d8aa6640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 0x7fd7d4153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d3fff640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 0x7fd7d4158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d37fe640 1 -- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 msgr2=0x7fd7d4153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.498+0000 7fd7d37fe640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 0x7fd7d4153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d37fe640 1 -- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 msgr2=0x7fd7d4158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d37fe640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 0x7fd7d4158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d37fe640 1 -- 172.21.15.70:0/2417835480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7d4159620 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d8aa6640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 0x7fd7d4153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d3fff640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 0x7fd7d4158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d37fe640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800c4d0 tx=0x7fd7c800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8018070 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7c80074e0 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7d4159900 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8004a20 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.499+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7d40cda70 con 0x7fd7d4072d80 2024-04-15T15:15:21.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.500+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7d40689e0 con 0x7fd7d4072d80 2024-04-15T15:15:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.500+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd7c801b070 con 0x7fd7d4072d80 2024-04-15T15:15:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.501+0000 7fd7d0ff9640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 0x7fd7c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.501+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd7c808b120 con 0x7fd7d4072d80 2024-04-15T15:15:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.501+0000 7fd7d3fff640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 0x7fd7c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.501+0000 7fd7d3fff640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 0x7fd7c006edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4007fd0 tx=0x7fd7c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.505+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd7c80592a0 con 0x7fd7d4072d80 2024-04-15T15:15:21.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.691+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd7d40b2aa0 con 0x7fd7d4072d80 2024-04-15T15:15:21.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.692+0000 7fd7d0ff9640 1 -- 172.21.15.70:0/2417835480 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7fd7c8058ba0 con 0x7fd7d4072d80 2024-04-15T15:15:21.694 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:15:21.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 msgr2=0x7fd7c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 0x7fd7c006edb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd7c4007fd0 tx=0x7fd7c4009040 comp rx=0 tx=0).stop 2024-04-15T15:15:21.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 msgr2=0x7fd7d4153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800c4d0 tx=0x7fd7c800c990 comp rx=0 tx=0).stop 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 shutdown_connections 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7d40d0da0 0x7fd7d4153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd7c006c8f0 0x7fd7c006edb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7d4072d80 0x7fd7d4153300 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 --2- 172.21.15.70:0/2417835480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7d4071d10 0x7fd7d4158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 >> 172.21.15.70:0/2417835480 conn(0x7fd7d406be00 msgr2=0x7fd7d40736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 shutdown_connections 2024-04-15T15:15:21.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.694+0000 7fd7da509640 1 -- 172.21.15.70:0/2417835480 wait complete. 2024-04-15T15:15:21.707 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:15:21.707 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:21.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.864+0000 7fcd17704640 1 Processor -- start 2024-04-15T15:15:21.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 -- start start 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1015c8d0 0x7fcd100cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd1009b0e0 0x7fcd100b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd100d0350 con 0x7fcd1009b650 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcd100b5b00 con 0x7fcd1009b0e0 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.865+0000 7fcd17704640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcd100ce100 con 0x7fcd1015c8d0 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd1009b0e0 0x7fcd100b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcd1009b0e0 0x7fcd100b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54752/0 (socket says 172.21.15.70:54752) 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 -- 172.21.15.70:0/3489578977 learned_addr learned my addr 172.21.15.70:0/3489578977 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd154a0640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1015c8d0 0x7fcd100cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 -- 172.21.15.70:0/3489578977 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1015c8d0 msgr2=0x7fcd100cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1015c8d0 0x7fcd100cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 -- 172.21.15.70:0/3489578977 >> v1:172.21.15.96:6789/0 conn(0x7fcd1009b650 legacy=0x7fcd1009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.866+0000 7fcd15ca1640 1 -- 172.21.15.70:0/3489578977 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd100b54b0 con 0x7fcd1009b0e0 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd15ca1640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009b0e0 0x7fcd100b5f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009870 tx=0x7fcd0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f6fe94efabd91cb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd0ffff640 1 -- 172.21.15.70:0/3489578977 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00035020 con 0x7fcd1009b0e0 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd0ffff640 1 -- 172.21.15.70:0/3489578977 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd00031640 con 0x7fcd1009b0e0 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd0ffff640 1 -- 172.21.15.70:0/3489578977 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0002d040 con 0x7fcd1009b0e0 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 -- 172.21.15.70:0/3489578977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009b0e0 msgr2=0x7fcd100b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009b0e0 0x7fcd100b5f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009870 tx=0x7fcd0002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 -- 172.21.15.70:0/3489578977 shutdown_connections 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009b0e0 0x7fcd100b5f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 --2- 172.21.15.70:0/3489578977 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1015c8d0 0x7fcd100cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 -- 172.21.15.70:0/3489578977 >> 172.21.15.70:0/3489578977 conn(0x7fcd100b3fb0 msgr2=0x7fcd100b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 -- 172.21.15.70:0/3489578977 shutdown_connections 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.867+0000 7fcd17704640 1 -- 172.21.15.70:0/3489578977 wait complete. 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 Processor -- start 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 -- start start 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd1009b0e0 0x7fcd1014f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1009b650 0x7fcd1009f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd100b6650 con 0x7fcd1009fad0 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcd100b64d0 con 0x7fcd1009b0e0 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd17704640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcd100b67d0 con 0x7fcd1009b650 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd15ca1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd15ca1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54760/0 (socket says 172.21.15.70:54760) 2024-04-15T15:15:21.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.869+0000 7fcd15ca1640 1 -- 172.21.15.70:0/4239741496 learned_addr learned my addr 172.21.15.70:0/4239741496 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd154a0640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd1009b0e0 0x7fcd1014f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd14c9f640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1009b650 0x7fcd1009f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 -- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1009b650 msgr2=0x7fcd1009f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1009b650 0x7fcd1009f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 -- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd1009b0e0 msgr2=0x7fcd1014f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd1009b0e0 0x7fcd1014f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 -- 172.21.15.70:0/4239741496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1014ffc0 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd15ca1640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcd0002b930 tx=0x7fcd000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00035040 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd000399f0 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd101502a0 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd00039dc0 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.870+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd10150860 con 0x7fcd1009fad0 2024-04-15T15:15:21.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.871+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd100690f0 con 0x7fcd1009fad0 2024-04-15T15:15:21.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.872+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcd00038020 con 0x7fcd1009fad0 2024-04-15T15:15:21.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.872+0000 7fcd0dffb640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 0x7fccfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:21.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.872+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcd000adae0 con 0x7fcd1009fad0 2024-04-15T15:15:21.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.872+0000 7fcd154a0640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 0x7fccfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:21.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.873+0000 7fcd154a0640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 0x7fccfc06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009aa0 tx=0x7fcd08009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:21.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:21.877+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcd0007b610 con 0x7fcd1009fad0 2024-04-15T15:15:22.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.056+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcd100cecb0 con 0x7fcd1009fad0 2024-04-15T15:15:22.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.057+0000 7fcd0dffb640 1 -- 172.21.15.70:0/4239741496 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7fcd0007f910 con 0x7fcd1009fad0 2024-04-15T15:15:22.059 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:22.060 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 msgr2=0x7fccfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 0x7fccfc06edb0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fcd08009aa0 tx=0x7fcd08009190 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 msgr2=0x7fcd10151060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcd0002b930 tx=0x7fcd000046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 shutdown_connections 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fccfc06c8f0 0x7fccfc06edb0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd1009fad0 0x7fcd10151060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcd1009b650 0x7fcd1009f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 --2- 172.21.15.70:0/4239741496 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcd1009b0e0 0x7fcd1014f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 >> 172.21.15.70:0/4239741496 conn(0x7fcd100b3fb0 msgr2=0x7fcd100b4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 shutdown_connections 2024-04-15T15:15:22.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.060+0000 7fcd17704640 1 -- 172.21.15.70:0/4239741496 wait complete. 2024-04-15T15:15:22.073 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:22.073 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:22.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.177+0000 7f253a3f4640 1 Processor -- start 2024-04-15T15:15:22.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.177+0000 7f253a3f4640 1 -- start start 2024-04-15T15:15:22.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f253a3f4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2534072d80 0x7f2534071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f253a3f4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f25340af3e0 0x7f25340af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f253a3f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253415db40 con 0x7f25340d0690 2024-04-15T15:15:22.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f253a3f4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f253415cfc0 con 0x7f2534072d80 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f253a3f4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f253415d2a0 con 0x7f25340af3e0 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f2533fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2534072d80 0x7f2534071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f2538991640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f25340d0690 0x7f253406e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:50830/0 (socket says 172.21.15.70:50830) 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.178+0000 7f2538991640 1 -- 172.21.15.70:0/1904963762 learned_addr learned my addr 172.21.15.70:0/1904963762 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f252bfff640 1 --2- 172.21.15.70:0/1904963762 >> v2:172.21.15.96:3300/0 conn(0x7f25340af3e0 0x7f25340af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f2533fff640 1 -- 172.21.15.70:0/1904963762 >> v2:172.21.15.96:3300/0 conn(0x7f25340af3e0 msgr2=0x7f25340af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f2533fff640 1 --2- 172.21.15.70:0/1904963762 >> v2:172.21.15.96:3300/0 conn(0x7f25340af3e0 0x7f25340af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f2533fff640 1 -- 172.21.15.70:0/1904963762 >> v1:172.21.15.70:6789/0 conn(0x7f25340d0690 legacy=0x7f253406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f25337fe640 1 -- 172.21.15.70:0/1904963762 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 682016299 0 0) 0x7f253415db40 con 0x7f25340d0690 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f2533fff640 1 -- 172.21.15.70:0/1904963762 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253415d580 con 0x7f2534072d80 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.179+0000 7f2533fff640 1 --2- 172.21.15.70:0/1904963762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f2534071d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f25340d1240 tx=0x7f252402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0f721791d946148 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f25337fe640 1 -- 172.21.15.70:0/1904963762 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25240315a0 con 0x7f2534072d80 2024-04-15T15:15:22.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f25337fe640 1 -- 172.21.15.70:0/1904963762 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2524037070 con 0x7f2534072d80 2024-04-15T15:15:22.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f25337fe640 1 -- 172.21.15.70:0/1904963762 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524031960 con 0x7f2534072d80 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 -- 172.21.15.70:0/1904963762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 msgr2=0x7f2534071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 --2- 172.21.15.70:0/1904963762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f2534071d10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f25340d1240 tx=0x7f252402b840 comp rx=0 tx=0).stop 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 -- 172.21.15.70:0/1904963762 shutdown_connections 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 --2- 172.21.15.70:0/1904963762 >> v2:172.21.15.96:3300/0 conn(0x7f25340af3e0 0x7f25340af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 --2- 172.21.15.70:0/1904963762 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f2534071d10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 -- 172.21.15.70:0/1904963762 >> 172.21.15.70:0/1904963762 conn(0x7f253406be00 msgr2=0x7f253406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 -- 172.21.15.70:0/1904963762 shutdown_connections 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.180+0000 7f253a3f4640 1 -- 172.21.15.70:0/1904963762 wait complete. 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.181+0000 7f253a3f4640 1 Processor -- start 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.181+0000 7f253a3f4640 1 -- start start 2024-04-15T15:15:22.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 0x7f2534159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f25340d0690 0x7f25341f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f253415cd10 con 0x7f2534072d80 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f253415cb90 con 0x7f25340af3e0 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f253a3f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f25340d0b00 con 0x7f25340d0690 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2533fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2533fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:54772/0 (socket says 172.21.15.70:54772) 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2533fff640 1 -- 172.21.15.70:0/3531884648 learned_addr learned my addr 172.21.15.70:0/3531884648 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2538991640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f25340d0690 0x7f25341f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.187 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f252bfff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 0x7f2534159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2533fff640 1 -- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f25340d0690 msgr2=0x7f25341f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.182+0000 7f2533fff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f25340d0690 0x7f25341f5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f2533fff640 1 -- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 msgr2=0x7f2534159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f2533fff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 0x7f2534159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f2533fff640 1 -- 172.21.15.70:0/3531884648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2534155240 con 0x7f2534072d80 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f252bfff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 0x7f2534159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f2533fff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f252402be50 tx=0x7f25240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524032630 con 0x7f2534072d80 2024-04-15T15:15:22.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25240327d0 con 0x7f2534072d80 2024-04-15T15:15:22.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2524032ba0 con 0x7f2534072d80 2024-04-15T15:15:22.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25340ce930 con 0x7f2534072d80 2024-04-15T15:15:22.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.183+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25340ceef0 con 0x7f2534072d80 2024-04-15T15:15:22.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.184+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25340689e0 con 0x7f2534072d80 2024-04-15T15:15:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.185+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f252404c020 con 0x7f2534072d80 2024-04-15T15:15:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.185+0000 7f25317fa640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.185+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f252402f020 con 0x7f2534072d80 2024-04-15T15:15:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.185+0000 7f252bfff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.185+0000 7f252bfff640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 0x7f251406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25340cfca0 tx=0x7f251c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.189+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2524040070 con 0x7f2534072d80 2024-04-15T15:15:22.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.370+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f25340ce670 con 0x7f2534072d80 2024-04-15T15:15:22.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.370+0000 7f25317fa640 1 -- 172.21.15.70:0/3531884648 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f252407bc90 con 0x7f2534072d80 2024-04-15T15:15:22.372 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:22.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 msgr2=0x7f251406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 0x7f251406edb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f25340cfca0 tx=0x7f251c008040 comp rx=0 tx=0).stop 2024-04-15T15:15:22.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 msgr2=0x7f253409a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f252402be50 tx=0x7f25240046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 shutdown_connections 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f25340d0690 0x7f25341f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f25340af3e0 0x7f2534159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f251406c8f0 0x7f251406edb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 --2- 172.21.15.70:0/3531884648 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2534072d80 0x7f253409a250 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 >> 172.21.15.70:0/3531884648 conn(0x7f253406be00 msgr2=0x7f25340b2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 shutdown_connections 2024-04-15T15:15:22.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.372+0000 7f253a3f4640 1 -- 172.21.15.70:0/3531884648 wait complete. 2024-04-15T15:15:22.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:22.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.491+0000 7f91b5196640 1 Processor -- start 2024-04-15T15:15:22.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.491+0000 7f91b5196640 1 -- start start 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.491+0000 7f91b5196640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b00d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91b5196640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f91b00d21e0 0x7f91b00d06b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91b5196640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0bf0 0x7f91b0160230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91b5196640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f91b00d1020 con 0x7f91b00d21e0 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91b5196640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f91b006b9e0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91b5196640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f91b006bc80 con 0x7f91b0153670 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f91b00d21e0 0x7f91b00d06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f91b00d21e0 0x7f91b00d06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36286/0 (socket says 172.21.15.70:36286) 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 -- 172.21.15.70:0/1690077067 learned_addr learned my addr 172.21.15.70:0/1690077067 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91af59e640 1 --2- 172.21.15.70:0/1690077067 >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0bf0 0x7f91b0160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91aed9d640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b00d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 -- 172.21.15.70:0/1690077067 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 msgr2=0x7f91b00d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b00d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 -- 172.21.15.70:0/1690077067 >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0bf0 msgr2=0x7f91b0160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 --2- 172.21.15.70:0/1690077067 >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0bf0 0x7f91b0160230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.492+0000 7f91ae59c640 1 -- 172.21.15.70:0/1690077067 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b006bf60 con 0x7f91b00d21e0 2024-04-15T15:15:22.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91ae59c640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d21e0 0x7f91b00d06b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f91b0072e60 tx=0x7f91a402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142d8721214f48e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91add9b640 1 -- 172.21.15.70:0/1690077067 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a402e070 con 0x7f91b00d21e0 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91add9b640 1 -- 172.21.15.70:0/1690077067 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a4037070 con 0x7f91b00d21e0 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91add9b640 1 -- 172.21.15.70:0/1690077067 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4031a00 con 0x7f91b00d21e0 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 -- 172.21.15.70:0/1690077067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d21e0 msgr2=0x7f91b00d06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d21e0 0x7f91b00d06b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f91b0072e60 tx=0x7f91a402b880 comp rx=0 tx=0).stop 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 -- 172.21.15.70:0/1690077067 shutdown_connections 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 --2- 172.21.15.70:0/1690077067 >> v2:172.21.15.96:3300/0 conn(0x7f91b00d0bf0 0x7f91b0160230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d21e0 0x7f91b00d06b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 --2- 172.21.15.70:0/1690077067 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b00d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.493+0000 7f91b5196640 1 -- 172.21.15.70:0/1690077067 >> 172.21.15.70:0/1690077067 conn(0x7f91b009a9d0 msgr2=0x7f91b00d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.494+0000 7f91b5196640 1 -- 172.21.15.70:0/1690077067 shutdown_connections 2024-04-15T15:15:22.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.494+0000 7f91b5196640 1 -- 172.21.15.70:0/1690077067 wait complete. 2024-04-15T15:15:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.495+0000 7f91b5196640 1 Processor -- start 2024-04-15T15:15:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.495+0000 7f91b5196640 1 -- start start 2024-04-15T15:15:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 0x7f91b00ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.504 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b0154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91b015d9c0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91b0153a50 con 0x7f91b00d21e0 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f91b015db40 con 0x7f91b0153670 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36288/0 (socket says 172.21.15.70:36288) 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 -- 172.21.15.70:0/978682399 learned_addr learned my addr 172.21.15.70:0/978682399 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91ae59c640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 0x7f91b00ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91af59e640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b0154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 -- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 msgr2=0x7f91b0154c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b0154c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 -- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 msgr2=0x7f91b00ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 0x7f91b00ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 -- 172.21.15.70:0/978682399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b014b4c0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91ae59c640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 0x7f91b00ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91af59e640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b0154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91aed9d640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91a000e9b0 tx=0x7f91a000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0017070 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a000c9a0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.496+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b0149260 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.497+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a000cd90 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.497+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b0149a60 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.498+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b00690f0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.498+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91a001a070 con 0x7f91b00d0bf0 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.498+0000 7f91977fe640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.506 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.499+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f91a008bea0 con 0x7f91b00d0bf0 2024-04-15T15:15:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.499+0000 7f91ae59c640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.499+0000 7f91ae59c640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 0x7f919c06edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f91a4036000 tx=0x7f91a403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.503+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91a0059f90 con 0x7f91b00d0bf0 2024-04-15T15:15:22.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.676+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f91b00b3b00 con 0x7f91b00d0bf0 2024-04-15T15:15:22.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.677+0000 7f91977fe640 1 -- 172.21.15.70:0/978682399 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f91a0059890 con 0x7f91b00d0bf0 2024-04-15T15:15:22.679 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 msgr2=0x7f919c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 0x7f919c06edb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f91a4036000 tx=0x7f91a403d040 comp rx=0 tx=0).stop 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 msgr2=0x7f91b00ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f91a000e9b0 tx=0x7f91a000ee70 comp rx=0 tx=0).stop 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 shutdown_connections 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91b0153670 0x7f91b0154c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91b00d21e0 0x7f91b00ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f919c06c8f0 0x7f919c06edb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 --2- 172.21.15.70:0/978682399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91b00d0bf0 0x7f91b00ce000 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 >> 172.21.15.70:0/978682399 conn(0x7f91b009a9d0 msgr2=0x7f91b00af470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 shutdown_connections 2024-04-15T15:15:22.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.679+0000 7f91b5196640 1 -- 172.21.15.70:0/978682399 wait complete. 2024-04-15T15:15:22.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:22.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.796+0000 7f3f57e77640 1 Processor -- start 2024-04-15T15:15:22.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 -- start start 2024-04-15T15:15:22.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 0x7f3f5006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3f5009a9d0 0x7f3f5009ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f5006e9d0 con 0x7f3f500b4ab0 2024-04-15T15:15:22.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3f5015ed70 con 0x7f3f5009a9d0 2024-04-15T15:15:22.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.797+0000 7f3f57e77640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3f5006e000 con 0x7f3f5015e990 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f56414640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3f5009a9d0 0x7f3f5009ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f55412640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3f500b4ab0 0x7f3f5009b720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44622/0 (socket says 172.21.15.70:44622) 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f55412640 1 -- 172.21.15.70:0/1203549959 learned_addr learned my addr 172.21.15.70:0/1203549959 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f55c13640 1 --2- 172.21.15.70:0/1203549959 >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 0x7f3f5006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f56414640 1 -- 172.21.15.70:0/1203549959 >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 msgr2=0x7f3f5006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f56414640 1 --2- 172.21.15.70:0/1203549959 >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 0x7f3f5006e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f56414640 1 -- 172.21.15.70:0/1203549959 >> v1:172.21.15.70:6789/0 conn(0x7f3f500b4ab0 legacy=0x7f3f5009b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f54c11640 1 -- 172.21.15.70:0/1203549959 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1652541821 0 0) 0x7f3f5006e9d0 con 0x7f3f500b4ab0 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f56414640 1 -- 172.21.15.70:0/1203549959 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5006d3f0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.798+0000 7f3f55c13640 1 --2- 172.21.15.70:0/1203549959 >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 0x7f3f5006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f56414640 1 --2- 172.21.15.70:0/1203549959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5009ae00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3f50073540 tx=0x7f3f4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c73bab1d1fe6279 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f54c11640 1 -- 172.21.15.70:0/1203549959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48039070 con 0x7f3f5009a9d0 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f54c11640 1 -- 172.21.15.70:0/1203549959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4802cc80 con 0x7f3f5009a9d0 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f54c11640 1 -- 172.21.15.70:0/1203549959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034660 con 0x7f3f5009a9d0 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 -- 172.21.15.70:0/1203549959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 msgr2=0x7f3f5009ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1203549959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5009ae00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3f50073540 tx=0x7f3f4802bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 -- 172.21.15.70:0/1203549959 shutdown_connections 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1203549959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5009ae00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1203549959 >> v2:172.21.15.96:3300/0 conn(0x7f3f5015e990 0x7f3f5006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 -- 172.21.15.70:0/1203549959 >> 172.21.15.70:0/1203549959 conn(0x7f3f500b3eb0 msgr2=0x7f3f500b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 -- 172.21.15.70:0/1203549959 shutdown_connections 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.799+0000 7f3f57e77640 1 -- 172.21.15.70:0/1203549959 wait complete. 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 Processor -- start 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 -- start start 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3f500b4ab0 0x7f3f500b6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f5015e990 0x7f3f500c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5006d9b0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3f5009b340 con 0x7f3f5015e990 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f57e77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3f5006dc90 con 0x7f3f500b4ab0 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f55c13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f55c13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36314/0 (socket says 172.21.15.70:36314) 2024-04-15T15:15:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.801+0000 7f3f55c13640 1 -- 172.21.15.70:0/1701684749 learned_addr learned my addr 172.21.15.70:0/1701684749 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 -- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3f500b4ab0 msgr2=0x7f3f500b6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3f500b4ab0 0x7f3f500b6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 -- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f5015e990 msgr2=0x7f3f500c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f5015e990 0x7f3f500c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 -- 172.21.15.70:0/1701684749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f500b52e0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f55c13640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00c7f0 tx=0x7f3f4c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c015070 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4c007a30 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c007e00 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f500b55c0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.802+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f500b5b80 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.803+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f500689e0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.803+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3f4c018070 con 0x7f3f5009a9d0 2024-04-15T15:15:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.804+0000 7f3f467fc640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 0x7f3f3406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.804+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3f4c08be10 con 0x7f3f5009a9d0 2024-04-15T15:15:22.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.804+0000 7f3f55412640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 0x7f3f3406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:22.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.804+0000 7f3f55412640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 0x7f3f3406edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007fd0 tx=0x7f3f40007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.808+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3f4c059f90 con 0x7f3f5009a9d0 2024-04-15T15:15:22.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.985+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3f500d0ab0 con 0x7f3f5009a9d0 2024-04-15T15:15:22.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.986+0000 7f3f467fc640 1 -- 172.21.15.70:0/1701684749 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f3f4c059890 con 0x7f3f5009a9d0 2024-04-15T15:15:22.988 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 msgr2=0x7f3f3406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 0x7f3f3406edb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3f40007fd0 tx=0x7f3f40007960 comp rx=0 tx=0).stop 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 msgr2=0x7f3f5014d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00c7f0 tx=0x7f3f4c00cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 shutdown_connections 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3f5015e990 0x7f3f500c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3f500b4ab0 0x7f3f500b6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3f3406c8f0 0x7f3f3406edb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 --2- 172.21.15.70:0/1701684749 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3f5009a9d0 0x7f3f5014d810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 >> 172.21.15.70:0/1701684749 conn(0x7f3f500b3eb0 msgr2=0x7f3f500cdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 shutdown_connections 2024-04-15T15:15:22.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:22.988+0000 7f3f57e77640 1 -- 172.21.15.70:0/1701684749 wait complete. 2024-04-15T15:15:23.001 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.106+0000 7fbfab37c640 1 Processor -- start 2024-04-15T15:15:23.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.106+0000 7fbfab37c640 1 -- start start 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfab37c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfa409a9d0 0x7fbfa4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfab37c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa415fb30 0x7fbfa40d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfab37c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa4070720 con 0x7fbfa40701f0 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfab37c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa409ba90 con 0x7fbfa409a9d0 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfab37c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa40d08c0 con 0x7fbfa415fb30 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfa409a9d0 0x7fbfa4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfa409a9d0 0x7fbfa4160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36330/0 (socket says 172.21.15.70:36330) 2024-04-15T15:15:23.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 -- 172.21.15.70:0/1076141156 learned_addr learned my addr 172.21.15.70:0/1076141156 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9919640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa415fb30 0x7fbfa40d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 -- 172.21.15.70:0/1076141156 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa415fb30 msgr2=0x7fbfa40d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa415fb30 0x7fbfa40d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 -- 172.21.15.70:0/1076141156 >> v1:172.21.15.96:6789/0 conn(0x7fbfa40701f0 legacy=0x7fbfa409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.107+0000 7fbfa9118640 1 -- 172.21.15.70:0/1076141156 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa4070a90 con 0x7fbfa409a9d0 2024-04-15T15:15:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.108+0000 7fbfa9118640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa409a9d0 0x7fbfa4160420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009870 tx=0x7fbfa002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3984fda02e3b3032 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.108+0000 7fbf9bfff640 1 -- 172.21.15.70:0/1076141156 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0034070 con 0x7fbfa409a9d0 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.108+0000 7fbf9bfff640 1 -- 172.21.15.70:0/1076141156 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0030840 con 0x7fbfa409a9d0 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.108+0000 7fbf9bfff640 1 -- 172.21.15.70:0/1076141156 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0030c10 con 0x7fbfa409a9d0 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 -- 172.21.15.70:0/1076141156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa409a9d0 msgr2=0x7fbfa4160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa409a9d0 0x7fbfa4160420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009870 tx=0x7fbfa002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 -- 172.21.15.70:0/1076141156 shutdown_connections 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa415fb30 0x7fbfa40d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 --2- 172.21.15.70:0/1076141156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa409a9d0 0x7fbfa4160420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 -- 172.21.15.70:0/1076141156 >> 172.21.15.70:0/1076141156 conn(0x7fbfa40d2460 msgr2=0x7fbfa40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 -- 172.21.15.70:0/1076141156 shutdown_connections 2024-04-15T15:15:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.109+0000 7fbfab37c640 1 -- 172.21.15.70:0/1076141156 wait complete. 2024-04-15T15:15:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.110+0000 7fbfab37c640 1 Processor -- start 2024-04-15T15:15:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.110+0000 7fbfab37c640 1 -- start start 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa409a9d0 0x7fbfa406dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415fb30 0x7fbfa415bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa415ffc0 con 0x7fbfa40701f0 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa4071050 con 0x7fbfa415fb30 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa40b1c60 con 0x7fbfa409a9d0 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36336/0 (socket says 172.21.15.70:36336) 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 -- 172.21.15.70:0/3425358767 learned_addr learned my addr 172.21.15.70:0/3425358767 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa8917640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa409a9d0 0x7fbfa406dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9919640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415fb30 0x7fbfa415bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 -- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa409a9d0 msgr2=0x7fbfa406dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa409a9d0 0x7fbfa406dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 -- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415fb30 msgr2=0x7fbfa415bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415fb30 0x7fbfa415bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 -- 172.21.15.70:0/3425358767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa415d3c0 con 0x7fbfa40701f0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfa9118640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0000c00 tx=0x7fbfa0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0048070 con 0x7fbfa40701f0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0030640 con 0x7fbfa40701f0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa415d6a0 con 0x7fbfa40701f0 2024-04-15T15:15:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.111+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0034040 con 0x7fbfa40701f0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.112+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa415df40 con 0x7fbfa40701f0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.113+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa415d870 con 0x7fbfa40701f0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.113+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfa004b070 con 0x7fbfa40701f0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.113+0000 7fbf99ffb640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 0x7fbf8806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.114+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbfa00adf10 con 0x7fbfa40701f0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.114+0000 7fbfa8917640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 0x7fbf8806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.114+0000 7fbfa8917640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 0x7fbf8806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40cee30 tx=0x7fbf94009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.118+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfa007bf70 con 0x7fbfa40701f0 2024-04-15T15:15:23.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.294+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fbfa415e180 con 0x7fbfa40701f0 2024-04-15T15:15:23.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.294+0000 7fbf99ffb640 1 -- 172.21.15.70:0/3425358767 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fbfa415e180 con 0x7fbfa40701f0 2024-04-15T15:15:23.297 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 msgr2=0x7fbf8806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 0x7fbf8806edb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbfa40cee30 tx=0x7fbf94009040 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 msgr2=0x7fbfa40c7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0000c00 tx=0x7fbfa0004730 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 shutdown_connections 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415fb30 0x7fbfa415bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa409a9d0 0x7fbfa406dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c8f0 0x7fbf8806edb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 --2- 172.21.15.70:0/3425358767 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40701f0 0x7fbfa40c7920 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.296+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 >> 172.21.15.70:0/3425358767 conn(0x7fbfa40d2460 msgr2=0x7fbfa4154940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.297+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 shutdown_connections 2024-04-15T15:15:23.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.297+0000 7fbfab37c640 1 -- 172.21.15.70:0/3425358767 wait complete. 2024-04-15T15:15:23.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.412+0000 7f375d50e640 1 Processor -- start 2024-04-15T15:15:23.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.412+0000 7f375d50e640 1 -- start start 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f375d50e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f375809a9d0 0x7f3758160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f375d50e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f375815fb30 0x7f37580d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f375d50e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3758070720 con 0x7f37580701f0 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f375d50e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f375809ba90 con 0x7f375815fb30 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f375d50e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37580d08c0 con 0x7f375809a9d0 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f375815fb30 0x7f37580d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f374ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f37580701f0 0x7f375809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44638/0 (socket says 172.21.15.70:44638) 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f374ffff640 1 -- 172.21.15.70:0/3410613910 learned_addr learned my addr 172.21.15.70:0/3410613910 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f3756ffd640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f375809a9d0 0x7f3758160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37577fe640 1 -- 172.21.15.70:0/3410613910 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f375809a9d0 msgr2=0x7f3758160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37577fe640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f375809a9d0 0x7f3758160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37577fe640 1 -- 172.21.15.70:0/3410613910 >> v1:172.21.15.70:6789/0 conn(0x7f37580701f0 legacy=0x7f375809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37577fe640 1 -- 172.21.15.70:0/3410613910 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3758070a90 con 0x7f375815fb30 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.413+0000 7f37567fc640 1 -- 172.21.15.70:0/3410613910 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3289515407 0 0) 0x7f3758070720 con 0x7f37580701f0 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.414+0000 7f37577fe640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375815fb30 0x7f37580d0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3750009dc0 tx=0x7f375002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3df34a11673b2067 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.414+0000 7f37567fc640 1 -- 172.21.15.70:0/3410613910 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750039070 con 0x7f375815fb30 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.414+0000 7f37567fc640 1 -- 172.21.15.70:0/3410613910 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375002cc40 con 0x7f375815fb30 2024-04-15T15:15:23.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.414+0000 7f37567fc640 1 -- 172.21.15.70:0/3410613910 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3750034440 con 0x7f375815fb30 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 -- 172.21.15.70:0/3410613910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375815fb30 msgr2=0x7f37580d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375815fb30 0x7f37580d0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3750009dc0 tx=0x7f375002be90 comp rx=0 tx=0).stop 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 -- 172.21.15.70:0/3410613910 shutdown_connections 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375815fb30 0x7f37580d0d50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 --2- 172.21.15.70:0/3410613910 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f375809a9d0 0x7f3758160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 -- 172.21.15.70:0/3410613910 >> 172.21.15.70:0/3410613910 conn(0x7f37580d2460 msgr2=0x7f37580d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 -- 172.21.15.70:0/3410613910 shutdown_connections 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.415+0000 7f375d50e640 1 -- 172.21.15.70:0/3410613910 wait complete. 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.416+0000 7f375d50e640 1 Processor -- start 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.416+0000 7f375d50e640 1 -- start start 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37580701f0 0x7f37581f47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375815fb30 0x7f37581e9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f375815ffc0 con 0x7f375809a9d0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3758071050 con 0x7f375815fb30 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f375d50e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37580b1c60 con 0x7f37580701f0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36352/0 (socket says 172.21.15.70:36352) 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 -- 172.21.15.70:0/2799394148 learned_addr learned my addr 172.21.15.70:0/2799394148 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f3756ffd640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37580701f0 0x7f37581f47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f37577fe640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375815fb30 0x7f37581e9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 -- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37580701f0 msgr2=0x7f37581f47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37580701f0 0x7f37581f47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 -- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375815fb30 msgr2=0x7f37581e9610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375815fb30 0x7f37581e9610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 -- 172.21.15.70:0/2799394148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37580b0300 con 0x7f375809a9d0 2024-04-15T15:15:23.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374ffff640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f374000c410 tx=0x7f374000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.417+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37400074e0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.418+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37400047e0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.418+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37580b05e0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.418+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3740004bd0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.418+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37580b0ba0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.419+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37580689e0 con 0x7f375809a9d0 2024-04-15T15:15:23.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.419+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3740011070 con 0x7f375809a9d0 2024-04-15T15:15:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.420+0000 7f374f7fe640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 0x7f373806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.420+0000 7f3756ffd640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 0x7f373806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.420+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f374008b910 con 0x7f375809a9d0 2024-04-15T15:15:23.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.420+0000 7f3756ffd640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 0x7f373806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3748009b40 tx=0x7f3748009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.424+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3740059440 con 0x7f375809a9d0 2024-04-15T15:15:23.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.616+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f37580d3370 con 0x7f375809a9d0 2024-04-15T15:15:23.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.616+0000 7f374f7fe640 1 -- 172.21.15.70:0/2799394148 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f374005d740 con 0x7f375809a9d0 2024-04-15T15:15:23.618 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 msgr2=0x7f373806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 0x7f373806edb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3748009b40 tx=0x7f3748009190 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 msgr2=0x7f37581e90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f374000c410 tx=0x7f374000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 shutdown_connections 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f375815fb30 0x7f37581e9610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.618+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f373806c8f0 0x7f373806edb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.619+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f375809a9d0 0x7f37581e90d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.619+0000 7f375d50e640 1 --2- 172.21.15.70:0/2799394148 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37580701f0 0x7f37581f47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.619+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 >> 172.21.15.70:0/2799394148 conn(0x7f37580d2460 msgr2=0x7f375815ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.619+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 shutdown_connections 2024-04-15T15:15:23.621 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.619+0000 7f375d50e640 1 -- 172.21.15.70:0/2799394148 wait complete. 2024-04-15T15:15:23.631 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:15:23.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-15T15:15:23.657 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi070.stderr:ceph-fuse[43796]: fuse finished with error 0 and tester_r 0 2024-04-15T15:15:23.659 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:23.698 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:15:23.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:23.698 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-15T15:15:23.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:23.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.913+0000 7fa5b85a8640 1 Processor -- start 2024-04-15T15:15:23.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.913+0000 7fa5b85a8640 1 -- start start 2024-04-15T15:15:23.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b85a8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa5b00701f0 0x7fa5b0160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b85a8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5b015fb30 0x7fa5b00d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b85a8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5b0070720 con 0x7fa5b009a9d0 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b85a8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b009ba90 con 0x7fa5b015fb30 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b85a8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa5b00d08c0 con 0x7fa5b00701f0 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5b015fb30 0x7fa5b00d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5b015fb30 0x7fa5b00d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36368/0 (socket says 172.21.15.70:36368) 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 -- 172.21.15.70:0/1555534822 learned_addr learned my addr 172.21.15.70:0/1555534822 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b5b43640 1 --2- 172.21.15.70:0/1555534822 >> v2:172.21.15.96:3300/0 conn(0x7fa5b00701f0 0x7fa5b0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 -- 172.21.15.70:0/1555534822 >> v2:172.21.15.96:3300/0 conn(0x7fa5b00701f0 msgr2=0x7fa5b0160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 --2- 172.21.15.70:0/1555534822 >> v2:172.21.15.96:3300/0 conn(0x7fa5b00701f0 0x7fa5b0160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 -- 172.21.15.70:0/1555534822 >> v1:172.21.15.96:6789/0 conn(0x7fa5b009a9d0 legacy=0x7fa5b009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.914+0000 7fa5b6b45640 1 -- 172.21.15.70:0/1555534822 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b0070a90 con 0x7fa5b015fb30 2024-04-15T15:15:23.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.915+0000 7fa5b6b45640 1 --2- 172.21.15.70:0/1555534822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b015fb30 0x7fa5b00d0d50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0072180 tx=0x7fa5a802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca3a02ce24379978 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.915+0000 7fa5b5342640 1 -- 172.21.15.70:0/1555534822 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802ca80 con 0x7fa5b015fb30 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.915+0000 7fa5b5342640 1 -- 172.21.15.70:0/1555534822 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a802cc20 con 0x7fa5b015fb30 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.915+0000 7fa5b5342640 1 -- 172.21.15.70:0/1555534822 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a80343f0 con 0x7fa5b015fb30 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 -- 172.21.15.70:0/1555534822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b015fb30 msgr2=0x7fa5b00d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/1555534822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b015fb30 0x7fa5b00d0d50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5b0072180 tx=0x7fa5a802bee0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 -- 172.21.15.70:0/1555534822 shutdown_connections 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/1555534822 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b015fb30 0x7fa5b00d0d50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/1555534822 >> v2:172.21.15.96:3300/0 conn(0x7fa5b00701f0 0x7fa5b0160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 -- 172.21.15.70:0/1555534822 >> 172.21.15.70:0/1555534822 conn(0x7fa5b00d2460 msgr2=0x7fa5b00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 -- 172.21.15.70:0/1555534822 shutdown_connections 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.916+0000 7fa5b85a8640 1 -- 172.21.15.70:0/1555534822 wait complete. 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.917+0000 7fa5b85a8640 1 Processor -- start 2024-04-15T15:15:23.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.917+0000 7fa5b85a8640 1 -- start start 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5b00701f0 0x7fa5b015e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 0x7fa5b00b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b015ffc0 con 0x7fa5b009a9d0 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5b0071050 con 0x7fa5b015fb30 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b85a8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa5b00b1c60 con 0x7fa5b00701f0 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36376/0 (socket says 172.21.15.70:36376) 2024-04-15T15:15:23.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 -- 172.21.15.70:0/3880181776 learned_addr learned my addr 172.21.15.70:0/3880181776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b6344640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5b00701f0 0x7fa5b015e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b6b45640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 0x7fa5b00b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 -- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5b00701f0 msgr2=0x7fa5b015e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5b00701f0 0x7fa5b015e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 -- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 msgr2=0x7fa5b00b73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 0x7fa5b00b73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 -- 172.21.15.70:0/3880181776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5b00b7b10 con 0x7fa5b009a9d0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b6b45640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 0x7fa5b00b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5b5b43640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c7f0 tx=0x7fa5a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.918+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0015070 con 0x7fa5b009a9d0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.919+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5a0007820 con 0x7fa5b009a9d0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.919+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5b00b7df0 con 0x7fa5b009a9d0 2024-04-15T15:15:23.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.919+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0007bd0 con 0x7fa5b009a9d0 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.919+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5b00c6d30 con 0x7fa5b009a9d0 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.920+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5b00689e0 con 0x7fa5b009a9d0 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.920+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5a0018070 con 0x7fa5b009a9d0 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.921+0000 7fa5a6ffd640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 0x7fa59406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.921+0000 7fa5b6344640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 0x7fa59406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:23.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.921+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa5a008bbc0 con 0x7fa5b009a9d0 2024-04-15T15:15:23.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.921+0000 7fa5b6344640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 0x7fa59406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac006fd0 tx=0x7fa5ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:23.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:23.925+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5a0059cb0 con 0x7fa5b009a9d0 2024-04-15T15:15:24.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.102+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa5b00d3370 con 0x7fa5b009a9d0 2024-04-15T15:15:24.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.103+0000 7fa5a6ffd640 1 -- 172.21.15.70:0/3880181776 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7fa5a00595b0 con 0x7fa5b009a9d0 2024-04-15T15:15:24.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:24.105 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:24.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 msgr2=0x7fa59406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 0x7fa59406edb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac006fd0 tx=0x7fa5ac008040 comp rx=0 tx=0).stop 2024-04-15T15:15:24.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 msgr2=0x7fa5b00b6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c7f0 tx=0x7fa5a000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 shutdown_connections 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5b015fb30 0x7fa5b00b73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa59406c8f0 0x7fa59406edb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5b009a9d0 0x7fa5b00b6e70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 --2- 172.21.15.70:0/3880181776 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5b00701f0 0x7fa5b015e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 >> 172.21.15.70:0/3880181776 conn(0x7fa5b00d2460 msgr2=0x7fa5b01609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 shutdown_connections 2024-04-15T15:15:24.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.105+0000 7fa5b85a8640 1 -- 172.21.15.70:0/3880181776 wait complete. 2024-04-15T15:15:24.119 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:24.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.221+0000 7ff1fcd3d640 1 Processor -- start 2024-04-15T15:15:24.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.221+0000 7ff1fcd3d640 1 -- start start 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff1f80b3160 0x7ff1f815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1f815ef70 0x7ff1f815f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f80d0fb0 con 0x7ff1f815ef70 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff1f815cb50 con 0x7ff1f80b3160 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1fcd3d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f815bfd0 con 0x7ff1f80d0690 2024-04-15T15:15:24.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1f6d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1f815ef70 0x7ff1f815f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1f6d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff1f815ef70 0x7ff1f815f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36392/0 (socket says 172.21.15.70:36392) 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/98964076 learned_addr learned my addr 172.21.15.70:0/98964076 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1f659c640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.222+0000 7ff1edd9b640 1 --2- 172.21.15.70:0/98964076 >> v2:172.21.15.96:3300/0 conn(0x7ff1f80b3160 0x7ff1f815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/98964076 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 msgr2=0x7ff1f80d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f80d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/98964076 >> v2:172.21.15.96:3300/0 conn(0x7ff1f80b3160 msgr2=0x7ff1f815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/98964076 >> v2:172.21.15.96:3300/0 conn(0x7ff1f80b3160 0x7ff1f815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/98964076 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f815c2b0 con 0x7ff1f815ef70 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815f3d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80d1270 tx=0x7ff1f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ce2a4d1740077d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f5d9b640 1 -- 172.21.15.70:0/98964076 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f002ca70 con 0x7ff1f815ef70 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f5d9b640 1 -- 172.21.15.70:0/98964076 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f002cc10 con 0x7ff1f815ef70 2024-04-15T15:15:24.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.223+0000 7ff1f5d9b640 1 -- 172.21.15.70:0/98964076 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f00343f0 con 0x7ff1f815ef70 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/98964076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 msgr2=0x7ff1f815f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815f3d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff1f80d1270 tx=0x7ff1f002bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/98964076 shutdown_connections 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815f3d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/98964076 >> v2:172.21.15.96:3300/0 conn(0x7ff1f80b3160 0x7ff1f815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/98964076 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f80d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/98964076 >> 172.21.15.70:0/98964076 conn(0x7ff1f806be00 msgr2=0x7ff1f806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/98964076 shutdown_connections 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.224+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/98964076 wait complete. 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.225+0000 7ff1fcd3d640 1 Processor -- start 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 -- start start 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1f80b3160 0x7ff1f814d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f815dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f80b4fb0 con 0x7ff1f815ef70 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f815c870 con 0x7ff1f80b3160 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1fcd3d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff1f815bbd0 con 0x7ff1f80d0690 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1f6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1f6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36406/0 (socket says 172.21.15.70:36406) 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.226+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/1666680183 learned_addr learned my addr 172.21.15.70:0/1666680183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:24.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1edd9b640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f815dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f659c640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1f80b3160 0x7ff1f814d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 msgr2=0x7ff1f815dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f815dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1f80b3160 msgr2=0x7ff1f814d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1f80b3160 0x7ff1f814d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 -- 172.21.15.70:0/1666680183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f81f4560 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1f6d9d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009f40 tx=0x7ff1f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0044020 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1f0040690 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f81f4820 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0009510 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.227+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f81f4de0 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.228+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f80689e0 con 0x7ff1f815ef70 2024-04-15T15:15:24.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.229+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1f0047020 con 0x7ff1f815ef70 2024-04-15T15:15:24.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.229+0000 7ff1ef7fe640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 0x7ff1d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.229+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff1f00aeae0 con 0x7ff1f815ef70 2024-04-15T15:15:24.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.229+0000 7ff1f659c640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 0x7ff1d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.230+0000 7ff1f659c640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 0x7ff1d806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.233+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1f004e030 con 0x7ff1f815ef70 2024-04-15T15:15:24.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.411+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff1f81660e0 con 0x7ff1f815ef70 2024-04-15T15:15:24.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.412+0000 7ff1ef7fe640 1 -- 172.21.15.70:0/1666680183 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7ff1f007ccf0 con 0x7ff1f815ef70 2024-04-15T15:15:24.414 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:24.414 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 msgr2=0x7ff1d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 0x7ff1d806edb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff1e8006fd0 tx=0x7ff1e8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 msgr2=0x7ff1f815e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009f40 tx=0x7ff1f0004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 shutdown_connections 2024-04-15T15:15:24.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1d806c8f0 0x7ff1d806edb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1f815ef70 0x7ff1f815e400 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1f80d0690 0x7ff1f815dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 --2- 172.21.15.70:0/1666680183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1f80b3160 0x7ff1f814d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 >> 172.21.15.70:0/1666680183 conn(0x7ff1f806be00 msgr2=0x7ff1f80ae280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 shutdown_connections 2024-04-15T15:15:24.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.414+0000 7ff1fcd3d640 1 -- 172.21.15.70:0/1666680183 wait complete. 2024-04-15T15:15:24.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:15:24.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.532+0000 7f14140fb640 1 Processor -- start 2024-04-15T15:15:24.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.532+0000 7f14140fb640 1 -- start start 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f14140fb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f140c15e990 0x7f140c06e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f14140fb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f140c09b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f14140fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f140c06e9d0 con 0x7f140c09a9d0 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f14140fb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f140c15ed70 con 0x7f140c15e990 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f14140fb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f140c06e000 con 0x7f140c0b4ab0 2024-04-15T15:15:24.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1412698640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f140c09a9d0 0x7f140c0cca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44650/0 (socket says 172.21.15.70:44650) 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1412698640 1 -- 172.21.15.70:0/2699674597 learned_addr learned my addr 172.21.15.70:0/2699674597 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1411696640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f140c09b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1411e97640 1 --2- 172.21.15.70:0/2699674597 >> v2:172.21.15.96:3300/0 conn(0x7f140c15e990 0x7f140c06e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1219655374 0 0) 0x7f140c06e9d0 con 0x7f140c09a9d0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13f8003520 con 0x7f140c09a9d0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4120024712 0 0) 0x7f13f8003520 con 0x7f140c09a9d0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 msgr2=0x7f140c09b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f140c09b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 >> v2:172.21.15.96:3300/0 conn(0x7f140c15e990 msgr2=0x7f140c06e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 --2- 172.21.15.70:0/2699674597 >> v2:172.21.15.96:3300/0 conn(0x7f140c15e990 0x7f140c06e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140c06d3f0 con 0x7f140c09a9d0 2024-04-15T15:15:24.538 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f14040019f0 con 0x7f140c09a9d0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 >> v1:172.21.15.70:6789/0 conn(0x7f140c09a9d0 legacy=0x7f140c0cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f80039b0 0x7f13f8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f13f8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f13f80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f140c15ed70 con 0x7f13f8004800 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f140c06e000 con 0x7f13f80039b0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f13f800b9c0 con 0x7f140c0b4ab0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1411696640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f13f8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.533+0000 7f1411e97640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f80039b0 0x7f13f8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1412698640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f13f80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 -- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 msgr2=0x7f13f80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0b4ab0 0x7f13f80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 -- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f80039b0 msgr2=0x7f13f8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13f80039b0 0x7f13f8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 -- 172.21.15.70:0/2699674597 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f8003520 con 0x7f13f8004800 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.534+0000 7f1411696640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f13f8004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1400009ac0 tx=0x7f13f8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28cb56b8d82827f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400039070 con 0x7f13f8004800 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140002c6b0 con 0x7f13f8004800 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f1410e95640 1 -- 172.21.15.70:0/2699674597 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1400009690 con 0x7f13f8004800 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 -- 172.21.15.70:0/2699674597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 msgr2=0x7f13f8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 --2- 172.21.15.70:0/2699674597 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f13f8004c80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1400009ac0 tx=0x7f13f8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f1412698640 1 -- 172.21.15.70:0/2699674597 reap_dead start 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 -- 172.21.15.70:0/2699674597 shutdown_connections 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 -- 172.21.15.70:0/2699674597 >> 172.21.15.70:0/2699674597 conn(0x7f140c0b3eb0 msgr2=0x7f140c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 -- 172.21.15.70:0/2699674597 shutdown_connections 2024-04-15T15:15:24.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.535+0000 7f14140fb640 1 -- 172.21.15.70:0/2699674597 wait complete. 2024-04-15T15:15:24.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.536+0000 7f14140fb640 1 Processor -- start 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.536+0000 7f14140fb640 1 -- start start 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0cbb20 0x7f140c0c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 0x7f140c158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f140c06dc90 con 0x7f13f8004800 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f140c06d9b0 con 0x7f140c0c7e10 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f14140fb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f140c0cd120 con 0x7f140c0cbb20 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f1411e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f1411e97640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36424/0 (socket says 172.21.15.70:36424) 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f1411e97640 1 -- 172.21.15.70:0/3636470144 learned_addr learned my addr 172.21.15.70:0/3636470144 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f1412698640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 0x7f140c158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.537+0000 7f1411696640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0cbb20 0x7f140c0c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 -- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0cbb20 msgr2=0x7f140c0c78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0cbb20 0x7f140c0c78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 -- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 msgr2=0x7f140c158a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 0x7f140c158a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 -- 172.21.15.70:0/3636470144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f140c0c86e0 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1412698640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 0x7f140c158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f1411e97640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f14080119c0 tx=0x7f1408011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800faf0 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140800fc90 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140c0b6c20 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408014440 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.538+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f140c0b71e0 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.539+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f140c0689e0 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.539+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1408012040 con 0x7f13f8004800 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.540+0000 7f13fe7fc640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 0x7f13f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:24.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.540+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f140808c4c0 con 0x7f13f8004800 2024-04-15T15:15:24.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.540+0000 7f1411696640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 0x7f13f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:24.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.540+0000 7f1411696640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 0x7f13f006edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1400036000 tx=0x7f1400030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:24.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.544+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f140c0c86e0 con 0x7f13f8004800 2024-04-15T15:15:24.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:24.757+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f140c0c81f0 con 0x7f13f8004800 2024-04-15T15:15:25.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.330+0000 7f13fe7fc640 1 -- 172.21.15.70:0/3636470144 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7f1408019030 con 0x7f13f8004800 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 msgr2=0x7f13f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 0x7f13f006edb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1400036000 tx=0x7f1400030000 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 msgr2=0x7f140c14d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f14080119c0 tx=0x7f1408011e80 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 shutdown_connections 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f140c0c7e10 0x7f140c158a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f140c0cbb20 0x7f140c0c78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f13f006c8f0 0x7f13f006edb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 --2- 172.21.15.70:0/3636470144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13f8004800 0x7f140c14d7a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.332+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 >> 172.21.15.70:0/3636470144 conn(0x7f140c0b3eb0 msgr2=0x7f140c0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.333+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 shutdown_connections 2024-04-15T15:15:25.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.333+0000 7f14140fb640 1 -- 172.21.15.70:0/3636470144 wait complete. 2024-04-15T15:15:25.347 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-15T15:15:25.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:15:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.454+0000 7f5e00da8640 1 Processor -- start 2024-04-15T15:15:25.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.454+0000 7f5e00da8640 1 -- start start 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5e00da8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dfc15b520 0x7f5dfc0ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5e00da8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5dfc0cfae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5e00da8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dfc0cbee0 con 0x7f5dfc0cee70 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5e00da8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5dfc0cc1c0 con 0x7f5dfc15b520 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5e00da8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5dfc0cc4a0 con 0x7f5dfc0cf650 2024-04-15T15:15:25.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df9d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5dfc0cee70 0x7f5dfc0ce6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44664/0 (socket says 172.21.15.70:44664) 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df9d9b640 1 -- 172.21.15.70:0/3495231789 learned_addr learned my addr 172.21.15.70:0/3495231789 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5dfad9d640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5dfc0cfae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/3495231789 >> v2:172.21.15.96:3300/0 conn(0x7f5dfc15b520 0x7f5dfc0ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 633579756 0 0) 0x7f5dfc0cbee0 con 0x7f5dfc0cee70 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5de8003520 con 0x7f5dfc0cee70 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4069139790 0 0) 0x7f5de8003520 con 0x7f5dfc0cee70 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 msgr2=0x7f5dfc0cfae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5dfc0cfae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 >> v2:172.21.15.96:3300/0 conn(0x7f5dfc15b520 msgr2=0x7f5dfc0ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 --2- 172.21.15.70:0/3495231789 >> v2:172.21.15.96:3300/0 conn(0x7f5dfc15b520 0x7f5dfc0ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc0cc780 con 0x7f5dfc0cee70 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f5de4001670 con 0x7f5dfc0cee70 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 >> v1:172.21.15.70:6789/0 conn(0x7f5dfc0cee70 legacy=0x7f5dfc0ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de80039b0 0x7f5de8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5de8004800 0x7f5de8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5de80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc0cc1c0 con 0x7f5de8004800 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc0cc4a0 con 0x7f5de80039b0 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5de800b9c0 con 0x7f5dfc0cf650 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5de8004800 0x7f5de8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de80039b0 0x7f5de8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.455+0000 7f5dfad9d640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5de80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 -- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 msgr2=0x7f5de80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc0cf650 0x7f5de80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 -- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de80039b0 msgr2=0x7f5de8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:25.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de80039b0 0x7f5de8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 -- 172.21.15.70:0/3495231789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de8003520 con 0x7f5de8004800 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5de8004800 0x7f5de8004c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5de4001050 tx=0x7f5de8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfbdf47021010be4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.456+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de4007d70 con 0x7f5de8004800 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5de4030070 con 0x7f5de8004800 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5df959a640 1 -- 172.21.15.70:0/3495231789 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de402cd40 con 0x7f5de8004800 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 -- 172.21.15.70:0/3495231789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5de8004800 msgr2=0x7f5de8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 --2- 172.21.15.70:0/3495231789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5de8004800 0x7f5de8004c80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5de4001050 tx=0x7f5de8005830 comp rx=0 tx=0).stop 2024-04-15T15:15:25.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5dfad9d640 1 -- 172.21.15.70:0/3495231789 reap_dead start 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 -- 172.21.15.70:0/3495231789 shutdown_connections 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 -- 172.21.15.70:0/3495231789 >> 172.21.15.70:0/3495231789 conn(0x7f5dfc0b2e70 msgr2=0x7f5dfc0b3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 -- 172.21.15.70:0/3495231789 shutdown_connections 2024-04-15T15:15:25.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.457+0000 7f5e00da8640 1 -- 172.21.15.70:0/3495231789 wait complete. 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.458+0000 7f5e00da8640 1 Processor -- start 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 -- start start 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 0x7f5dfc06bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 0x7f5dfc164430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc15b900 con 0x7f5dfc159fb0 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc0cfa60 con 0x7f5de8004800 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5e00da8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5dfc0c6b70 con 0x7f5dfc166240 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5dfad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36440/0 (socket says 172.21.15.70:36440) 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.459+0000 7f5dfad9d640 1 -- 172.21.15.70:0/1207866678 learned_addr learned my addr 172.21.15.70:0/1207866678 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 0x7f5dfc164430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 0x7f5dfc06bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 -- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 msgr2=0x7f5dfc164430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 0x7f5dfc164430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 -- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 msgr2=0x7f5dfc06bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 0x7f5dfc06bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 -- 172.21.15.70:0/1207866678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc0c6de0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5df9d9b640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 0x7f5dfc164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:25.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 0x7f5dfc06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5dfad9d640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc073b00 tx=0x7f5dec011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec004240 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dec01b070 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc0c70c0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec00f7a0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.460+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc0c7680 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.461+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dfc0690f0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.461+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5dec012040 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.462+0000 7f5df2ffd640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.462+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5dec08bbd0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.462+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:25.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.462+0000 7f5dfa59c640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5df4007440 tx=0x7f5df400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:25.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.466+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5dec059cc0 con 0x7f5dfc159fb0 2024-04-15T15:15:25.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:25.680+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5dfc072fe0 con 0x7f5dfc159fb0 2024-04-15T15:15:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.337+0000 7f5df2ffd640 1 -- 172.21.15.70:0/1207866678 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7f5dec0595c0 con 0x7f5dfc159fb0 2024-04-15T15:15:26.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 msgr2=0x7f5ddc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5df4007440 tx=0x7f5df400b040 comp rx=0 tx=0).stop 2024-04-15T15:15:26.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 msgr2=0x7f5dfc164970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5dfc073b00 tx=0x7f5dec011cf0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 shutdown_connections 2024-04-15T15:15:26.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5ddc06c8f0 0x7f5ddc06edb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dfc159fb0 0x7f5dfc164970 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dfc166240 0x7f5dfc164430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 --2- 172.21.15.70:0/1207866678 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5de8004800 0x7f5dfc06bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 >> 172.21.15.70:0/1207866678 conn(0x7f5dfc0b2e70 msgr2=0x7f5dfc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:26.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 shutdown_connections 2024-04-15T15:15:26.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.340+0000 7f5e00da8640 1 -- 172.21.15.70:0/1207866678 wait complete. 2024-04-15T15:15:26.353 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:15:26.353 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 Processor -- start 2024-04-15T15:15:26.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 -- start start 2024-04-15T15:15:26.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 0x7f5d2009adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5d2015ebb0 0x7f5d2006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 0x7f5d200b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5d200cd8c0 con 0x7f5d2015ebb0 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5d2009b2f0 con 0x7f5d2009a9d0 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.456+0000 7f5d255f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5d200cccb0 con 0x7f5d2009b780 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5d2015ebb0 0x7f5d2006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5d2015ebb0 0x7f5d2006cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36444/0 (socket says 172.21.15.70:36444) 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 -- 172.21.15.70:0/3807205304 learned_addr learned my addr 172.21.15.70:0/3807205304 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d1f7fe640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 0x7f5d200b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d1effd640 1 --2- 172.21.15.70:0/3807205304 >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 0x7f5d2009adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 -- 172.21.15.70:0/3807205304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 msgr2=0x7f5d200b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 0x7f5d200b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 -- 172.21.15.70:0/3807205304 >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 msgr2=0x7f5d2009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:26.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 --2- 172.21.15.70:0/3807205304 >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 0x7f5d2009adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d17fff640 1 -- 172.21.15.70:0/3807205304 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d200ccf90 con 0x7f5d2015ebb0 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d1f7fe640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 0x7f5d200b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.457+0000 7f5d1effd640 1 --2- 172.21.15.70:0/3807205304 >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 0x7f5d2009adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d17fff640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2015ebb0 0x7f5d2006cd00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5d20072550 tx=0x7f5d0802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e2ffadf073f648c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d1e7fc640 1 -- 172.21.15.70:0/3807205304 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08035020 con 0x7f5d2015ebb0 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d1e7fc640 1 -- 172.21.15.70:0/3807205304 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d08031870 con 0x7f5d2015ebb0 2024-04-15T15:15:26.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d1e7fc640 1 -- 172.21.15.70:0/3807205304 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d08031c40 con 0x7f5d2015ebb0 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 -- 172.21.15.70:0/3807205304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2015ebb0 msgr2=0x7f5d2006cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2015ebb0 0x7f5d2006cd00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5d20072550 tx=0x7f5d0802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 -- 172.21.15.70:0/3807205304 shutdown_connections 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2009b780 0x7f5d200b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 --2- 172.21.15.70:0/3807205304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2015ebb0 0x7f5d2006cd00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 --2- 172.21.15.70:0/3807205304 >> v2:172.21.15.96:3300/0 conn(0x7f5d2009a9d0 0x7f5d2009adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 -- 172.21.15.70:0/3807205304 >> 172.21.15.70:0/3807205304 conn(0x7f5d200b3fb0 msgr2=0x7f5d200b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 -- 172.21.15.70:0/3807205304 shutdown_connections 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.458+0000 7f5d255f4640 1 -- 172.21.15.70:0/3807205304 wait complete. 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 Processor -- start 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 -- start start 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2009b780 0x7f5d200b7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 0x7f5d2015c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5d2006d240 con 0x7f5d2009a9d0 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d200cd550 con 0x7f5d2009b780 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d255f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5d200b5310 con 0x7f5d2015c350 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d1effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d1effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36446/0 (socket says 172.21.15.70:36446) 2024-04-15T15:15:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.460+0000 7f5d1effd640 1 -- 172.21.15.70:0/696419423 learned_addr learned my addr 172.21.15.70:0/696419423 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d17fff640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2009b780 0x7f5d200b7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1f7fe640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 0x7f5d2015c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 -- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 msgr2=0x7f5d2015c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 0x7f5d2015c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 -- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2009b780 msgr2=0x7f5d200b7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2009b780 0x7f5d200b7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 -- 172.21.15.70:0/696419423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d200b82c0 con 0x7f5d2009a9d0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1f7fe640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 0x7f5d2015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d1effd640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000c820 tx=0x7f5d1000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10015070 con 0x7f5d2009a9d0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d10007a30 con 0x7f5d2009a9d0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10007e00 con 0x7f5d2009a9d0 2024-04-15T15:15:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d200caa60 con 0x7f5d2009a9d0 2024-04-15T15:15:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.461+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d200cb020 con 0x7f5d2009a9d0 2024-04-15T15:15:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.462+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2015cd10 con 0x7f5d2009a9d0 2024-04-15T15:15:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.462+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5d10018070 con 0x7f5d2009a9d0 2024-04-15T15:15:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.463+0000 7f5d177fe640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.463+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5d1008bc40 con 0x7f5d2009a9d0 2024-04-15T15:15:26.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.463+0000 7f5d17fff640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.463+0000 7f5d17fff640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 0x7f5d0006ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d08036000 tx=0x7f5d0803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.467+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d200b82c0 con 0x7f5d2009a9d0 2024-04-15T15:15:26.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.679+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d200b5130 con 0x7f5d2009a9d0 2024-04-15T15:15:26.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.680+0000 7f5d177fe640 1 -- 172.21.15.70:0/696419423 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f5d200b5130 con 0x7f5d2009a9d0 2024-04-15T15:15:26.682 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:26.682 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:26.682 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:15:26.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 msgr2=0x7f5d0006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 0x7f5d0006ede0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d08036000 tx=0x7f5d0803c040 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 msgr2=0x7f5d20151a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000c820 tx=0x7f5d1000cc20 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 shutdown_connections 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d2015c350 0x7f5d2015c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d2009b780 0x7f5d200b7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d0006c920 0x7f5d0006ede0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.681+0000 7f5d255f4640 1 --2- 172.21.15.70:0/696419423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d2009a9d0 0x7f5d20151a70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.682+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 >> 172.21.15.70:0/696419423 conn(0x7f5d200b3fb0 msgr2=0x7f5d20160fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.682+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 shutdown_connections 2024-04-15T15:15:26.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.682+0000 7f5d255f4640 1 -- 172.21.15.70:0/696419423 wait complete. 2024-04-15T15:15:26.695 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:26.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.796+0000 7ff7ebf00640 1 Processor -- start 2024-04-15T15:15:26.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.796+0000 7ff7ebf00640 1 -- start start 2024-04-15T15:15:26.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7ebf00640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7e4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7ebf00640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e4070720 con 0x7ff7e409a9d0 2024-04-15T15:15:26.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7ebf00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7e4070a90 con 0x7ff7e415fb30 2024-04-15T15:15:26.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7ebf00640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e409ba90 con 0x7ff7e40701f0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7e9c9c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff7e409a9d0 0x7ff7e409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44670/0 (socket says 172.21.15.70:44670) 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7e9c9c640 1 -- 172.21.15.70:0/1255580788 learned_addr learned my addr 172.21.15.70:0/1255580788 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7e949b640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7e4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169357803 0 0) 0x7ff7e4070720 con 0x7ff7e409a9d0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.797+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d0003520 con 0x7ff7e409a9d0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 444312718 0 0) 0x7ff7d0003520 con 0x7ff7e409a9d0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 msgr2=0x7ff7e4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7e4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 >> v1:172.21.15.96:6789/0 conn(0x7ff7e415fb30 legacy=0x7ff7e40d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e4070d70 con 0x7ff7e409a9d0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7ff7e00019f0 con 0x7ff7e409a9d0 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 >> v1:172.21.15.70:6789/0 conn(0x7ff7e409a9d0 legacy=0x7ff7e409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7d00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d0003f60 0x7ff7d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0004d40 0x7ff7d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d000ba70 con 0x7ff7d0003f60 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d000bc20 con 0x7ff7d0004d40 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d000be20 con 0x7ff7e40701f0 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d0003f60 0x7ff7d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e9c9c640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7d00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7ea49d640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0004d40 0x7ff7d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 -- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 msgr2=0x7ff7d00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40701f0 0x7ff7d00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 -- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0004d40 msgr2=0x7ff7d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0004d40 0x7ff7d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7e949b640 1 -- 172.21.15.70:0/1255580788 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d0003520 con 0x7ff7d0003f60 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.798+0000 7ff7ea49d640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0004d40 0x7ff7d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:26.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7e949b640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d0003f60 0x7ff7d0004800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009a90 tx=0x7ff7d0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901219e2687cc24c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4039070 con 0x7ff7d0003f60 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7d402c680 con 0x7ff7d0003f60 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7e8c9a640 1 -- 172.21.15.70:0/1255580788 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7d4009690 con 0x7ff7d0003f60 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7ebf00640 1 -- 172.21.15.70:0/1255580788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d0003f60 msgr2=0x7ff7d0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/1255580788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7d0003f60 0x7ff7d0004800 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff7d4009a90 tx=0x7ff7d0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7ea49d640 1 -- 172.21.15.70:0/1255580788 reap_dead start 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7ebf00640 1 -- 172.21.15.70:0/1255580788 shutdown_connections 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.799+0000 7ff7ebf00640 1 -- 172.21.15.70:0/1255580788 >> 172.21.15.70:0/1255580788 conn(0x7ff7e40d2460 msgr2=0x7ff7e40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.800+0000 7ff7ebf00640 1 -- 172.21.15.70:0/1255580788 shutdown_connections 2024-04-15T15:15:26.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.800+0000 7ff7ebf00640 1 -- 172.21.15.70:0/1255580788 wait complete. 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 Processor -- start 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 -- start start 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0003f60 0x7ff7e4162630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 0x7ff7e40b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e415ffc0 con 0x7ff7e4149f90 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff7e40d08c0 con 0x7ff7d0003f60 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.801+0000 7ff7ebf00640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff7e40b1c60 con 0x7ff7e40b4ed0 2024-04-15T15:15:26.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36458/0 (socket says 172.21.15.70:36458) 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 -- 172.21.15.70:0/3370253235 learned_addr learned my addr 172.21.15.70:0/3370253235 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7e949b640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 0x7ff7e40b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 -- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 msgr2=0x7ff7e40b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7e9c9c640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0003f60 0x7ff7e4162630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 0x7ff7e40b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 -- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0003f60 msgr2=0x7ff7e4162630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0003f60 0x7ff7e4162630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 -- 172.21.15.70:0/3370253235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7e40b6b00 con 0x7ff7e4149f90 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7e949b640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 0x7ff7e40b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ea49d640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7e415ff10 tx=0x7ff7dc00e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc019070 con 0x7ff7e4149f90 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7dc0052e0 con 0x7ff7e4149f90 2024-04-15T15:15:26.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7e40b6da0 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7dc003830 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.802+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7e40b7360 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.803+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7e40689e0 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.804+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff7dc01c070 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.804+0000 7ff7da7fc640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 0x7ff7c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.804+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff7dc08b2c0 con 0x7ff7e4149f90 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.804+0000 7ff7e9c9c640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 0x7ff7c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:26.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.805+0000 7ff7e9c9c640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 0x7ff7c806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7e00013d0 tx=0x7ff7e0006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:26.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.808+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff7dc0593b0 con 0x7ff7e4149f90 2024-04-15T15:15:26.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.989+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7e4163440 con 0x7ff7e4149f90 2024-04-15T15:15:26.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.991+0000 7ff7da7fc640 1 -- 172.21.15.70:0/3370253235 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7ff7dc08b0e0 con 0x7ff7e4149f90 2024-04-15T15:15:26.993 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:26.993 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 msgr2=0x7ff7c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 0x7ff7c806edb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff7e00013d0 tx=0x7ff7e0006040 comp rx=0 tx=0).stop 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 msgr2=0x7ff7e414a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff7e415ff10 tx=0x7ff7dc00e820 comp rx=0 tx=0).stop 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 shutdown_connections 2024-04-15T15:15:26.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff7c806c8f0 0x7ff7c806edb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff7e4149f90 0x7ff7e414a410 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff7e40b4ed0 0x7ff7e40b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 --2- 172.21.15.70:0/3370253235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff7d0003f60 0x7ff7e4162630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 >> 172.21.15.70:0/3370253235 conn(0x7ff7e40d2460 msgr2=0x7ff7e406caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 shutdown_connections 2024-04-15T15:15:26.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:26.993+0000 7ff7ebf00640 1 -- 172.21.15.70:0/3370253235 wait complete. 2024-04-15T15:15:27.006 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:15:27.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:27.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.108+0000 7f7986da8640 1 Processor -- start 2024-04-15T15:15:27.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7986da8640 1 -- start start 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7986da8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f798015ebb0 0x7f798006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7986da8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79800ced60 con 0x7f798009b780 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7986da8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79800cccb0 con 0x7f798009a9d0 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7986da8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f79800cd8c0 con 0x7f798015ebb0 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f797ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f798015ebb0 0x7f798006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7985345640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f798009b780 0x7f79800b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44682/0 (socket says 172.21.15.70:44682) 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.109+0000 7f7985345640 1 -- 172.21.15.70:0/3426600249 learned_addr learned my addr 172.21.15.70:0/3426600249 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.110+0000 7f797ffff640 1 -- 172.21.15.70:0/3426600249 >> v1:172.21.15.96:6789/0 conn(0x7f798009a9d0 legacy=0x7f798009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.110+0000 7f797ffff640 1 -- 172.21.15.70:0/3426600249 >> v1:172.21.15.70:6789/0 conn(0x7f798009b780 legacy=0x7f79800b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.110+0000 7f797ffff640 1 -- 172.21.15.70:0/3426600249 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79800ccf90 con 0x7f798015ebb0 2024-04-15T15:15:27.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.110+0000 7f797f7fe640 1 -- 172.21.15.70:0/3426600249 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941994814 0 0) 0x7f7978001110 con 0x7f798009b780 2024-04-15T15:15:27.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.110+0000 7f797ffff640 1 --2- 172.21.15.70:0/3426600249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798015ebb0 0x7f798006cd00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f79680097a0 tx=0x7f796802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63535e645f8fe29 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f797f7fe640 1 -- 172.21.15.70:0/3426600249 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968034070 con 0x7f798015ebb0 2024-04-15T15:15:27.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f797f7fe640 1 -- 172.21.15.70:0/3426600249 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7968030610 con 0x7f798015ebb0 2024-04-15T15:15:27.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f797f7fe640 1 -- 172.21.15.70:0/3426600249 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79680309e0 con 0x7f798015ebb0 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 -- 172.21.15.70:0/3426600249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798015ebb0 msgr2=0x7f798006cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 --2- 172.21.15.70:0/3426600249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798015ebb0 0x7f798006cd00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f79680097a0 tx=0x7f796802b960 comp rx=0 tx=0).stop 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 -- 172.21.15.70:0/3426600249 shutdown_connections 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 --2- 172.21.15.70:0/3426600249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798015ebb0 0x7f798006cd00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 -- 172.21.15.70:0/3426600249 >> 172.21.15.70:0/3426600249 conn(0x7f79800b3fb0 msgr2=0x7f79800b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 -- 172.21.15.70:0/3426600249 shutdown_connections 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.111+0000 7f7986da8640 1 -- 172.21.15.70:0/3426600249 wait complete. 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.112+0000 7f7986da8640 1 Processor -- start 2024-04-15T15:15:27.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.112+0000 7f7986da8640 1 -- start start 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f798009b780 0x7f798015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7980157560 0x7f79801579d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f798006d240 con 0x7f798009a9d0 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79800cd550 con 0x7f798009b780 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7986da8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f79800b5310 con 0x7f7980157560 2024-04-15T15:15:27.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7984b44640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7984b44640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36472/0 (socket says 172.21.15.70:36472) 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.113+0000 7f7984b44640 1 -- 172.21.15.70:0/3437587838 learned_addr learned my addr 172.21.15.70:0/3437587838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 -- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7980157560 msgr2=0x7f79801579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7985345640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7980157560 0x7f79801579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7980157560 0x7f79801579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 -- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f798009b780 msgr2=0x7f798015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f798009b780 0x7f798015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 -- 172.21.15.70:0/3437587838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7980157f10 con 0x7f798009a9d0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7984b44640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f797000c870 tx=0x7f797000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7970003b70 con 0x7f798009a9d0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7970003d30 con 0x7f798009a9d0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79800b7490 con 0x7f798009a9d0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f797000aa90 con 0x7f798009a9d0 2024-04-15T15:15:27.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.114+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f798015f1f0 con 0x7f798009a9d0 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.115+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f798015f630 con 0x7f798009a9d0 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.115+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f797000ac30 con 0x7f798009a9d0 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.116+0000 7f797d7fa640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.116+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f797008bdf0 con 0x7f798009a9d0 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.116+0000 7f797ffff640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.116+0000 7f797ffff640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 0x7f796006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7968036000 tx=0x7f7968030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.120+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7970059f70 con 0x7f798009a9d0 2024-04-15T15:15:27.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.291+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f79800690f0 con 0x7f796006c8f0 2024-04-15T15:15:27.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.300+0000 7f797d7fa640 1 -- 172.21.15.70:0/3437587838 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f79800690f0 con 0x7f796006c8f0 2024-04-15T15:15:27.306 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:27.308 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 msgr2=0x7f796006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 0x7f796006edb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7968036000 tx=0x7f7968030000 comp rx=0 tx=0).stop 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 msgr2=0x7f79800cbca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f797000c870 tx=0x7f797000cd30 comp rx=0 tx=0).stop 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 shutdown_connections 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7980157560 0x7f79801579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f798009b780 0x7f798015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f796006c8f0 0x7f796006edb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.303+0000 7f7986da8640 1 --2- 172.21.15.70:0/3437587838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f798009a9d0 0x7f79800cbca0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.304+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 >> 172.21.15.70:0/3437587838 conn(0x7f79800b3fb0 msgr2=0x7f79800d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.304+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 shutdown_connections 2024-04-15T15:15:27.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.304+0000 7f7986da8640 1 -- 172.21.15.70:0/3437587838 wait complete. 2024-04-15T15:15:27.318 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-15T15:15:26.073286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236128,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738592,"statfs":{"total":773094113280,"available":772852318208,"internally_reserved":0,"allocated":3383296,"data_stored":2026910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.766019"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865551+0000","last_change":"2024-04-15T15:15:09.296489+0000","last_active":"2024-04-15T15:15:15.865551+0000","last_peered":"2024-04-15T15:15:15.865551+0000","last_clean":"2024-04-15T15:15:15.865551+0000","last_became_active":"2024-04-15T15:15:09.296225+0000","last_became_peered":"2024-04-15T15:15:09.296225+0000","last_unstale":"2024-04-15T15:15:15.865551+0000","last_undegraded":"2024-04-15T15:15:15.865551+0000","last_fullsized":"2024-04-15T15:15:15.865551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308730+0000","last_change":"2024-04-15T15:15:10.297175+0000","last_active":"2024-04-15T15:15:15.308730+0000","last_peered":"2024-04-15T15:15:15.308730+0000","last_clean":"2024-04-15T15:15:15.308730+0000","last_became_active":"2024-04-15T15:15:10.296707+0000","last_became_peered":"2024-04-15T15:15:10.296707+0000","last_unstale":"2024-04-15T15:15:15.308730+0000","last_undegraded":"2024-04-15T15:15:15.308730+0000","last_fullsized":"2024-04-15T15:15:15.308730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308030+0000","last_change":"2024-04-15T15:15:14.490094+0000","last_active":"2024-04-15T15:15:15.308030+0000","last_peered":"2024-04-15T15:15:15.308030+0000","last_clean":"2024-04-15T15:15:15.308030+0000","last_became_active":"2024-04-15T15:15:14.489727+0000","last_became_peered":"2024-04-15T15:15:14.489727+0000","last_unstale":"2024-04-15T15:15:15.308030+0000","last_undegraded":"2024-04-15T15:15:15.308030+0000","last_fullsized":"2024-04-15T15:15:15.308030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308822+0000","last_change":"2024-04-15T15:15:09.279298+0000","last_active":"2024-04-15T15:15:15.308822+0000","last_peered":"2024-04-15T15:15:15.308822+0000","last_clean":"2024-04-15T15:15:15.308822+0000","last_became_active":"2024-04-15T15:15:09.279018+0000","last_became_peered":"2024-04-15T15:15:09.279018+0000","last_unstale":"2024-04-15T15:15:15.308822+0000","last_undegraded":"2024-04-15T15:15:15.308822+0000","last_fullsized":"2024-04-15T15:15:15.308822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865283+0000","last_change":"2024-04-15T15:15:10.300836+0000","last_active":"2024-04-15T15:15:15.865283+0000","last_peered":"2024-04-15T15:15:15.865283+0000","last_clean":"2024-04-15T15:15:15.865283+0000","last_became_active":"2024-04-15T15:15:10.300574+0000","last_became_peered":"2024-04-15T15:15:10.300574+0000","last_unstale":"2024-04-15T15:15:15.865283+0000","last_undegraded":"2024-04-15T15:15:15.865283+0000","last_fullsized":"2024-04-15T15:15:15.865283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310208+0000","last_change":"2024-04-15T15:15:14.327665+0000","last_active":"2024-04-15T15:15:15.310208+0000","last_peered":"2024-04-15T15:15:15.310208+0000","last_clean":"2024-04-15T15:15:15.310208+0000","last_became_active":"2024-04-15T15:15:14.327161+0000","last_became_peered":"2024-04-15T15:15:14.327161+0000","last_unstale":"2024-04-15T15:15:15.310208+0000","last_undegraded":"2024-04-15T15:15:15.310208+0000","last_fullsized":"2024-04-15T15:15:15.310208+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866087+0000","last_change":"2024-04-15T15:15:09.297431+0000","last_active":"2024-04-15T15:15:15.866087+0000","last_peered":"2024-04-15T15:15:15.866087+0000","last_clean":"2024-04-15T15:15:15.866087+0000","last_became_active":"2024-04-15T15:15:09.297182+0000","last_became_peered":"2024-04-15T15:15:09.297182+0000","last_unstale":"2024-04-15T15:15:15.866087+0000","last_undegraded":"2024-04-15T15:15:15.866087+0000","last_fullsized":"2024-04-15T15:15:15.866087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:45.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865971+0000","last_change":"2024-04-15T15:15:10.303063+0000","last_active":"2024-04-15T15:15:15.865971+0000","last_peered":"2024-04-15T15:15:15.865971+0000","last_clean":"2024-04-15T15:15:15.865971+0000","last_became_active":"2024-04-15T15:15:10.302785+0000","last_became_peered":"2024-04-15T15:15:10.302785+0000","last_unstale":"2024-04-15T15:15:15.865971+0000","last_undegraded":"2024-04-15T15:15:15.865971+0000","last_fullsized":"2024-04-15T15:15:15.865971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309364+0000","last_change":"2024-04-15T15:15:14.328804+0000","last_active":"2024-04-15T15:15:15.309364+0000","last_peered":"2024-04-15T15:15:15.309364+0000","last_clean":"2024-04-15T15:15:15.309364+0000","last_became_active":"2024-04-15T15:15:14.327886+0000","last_became_peered":"2024-04-15T15:15:14.327886+0000","last_unstale":"2024-04-15T15:15:15.309364+0000","last_undegraded":"2024-04-15T15:15:15.309364+0000","last_fullsized":"2024-04-15T15:15:15.309364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:25.064499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865605+0000","last_change":"2024-04-15T15:15:09.300461+0000","last_active":"2024-04-15T15:15:15.865605+0000","last_peered":"2024-04-15T15:15:15.865605+0000","last_clean":"2024-04-15T15:15:15.865605+0000","last_became_active":"2024-04-15T15:15:09.300159+0000","last_became_peered":"2024-04-15T15:15:09.300159+0000","last_unstale":"2024-04-15T15:15:15.865605+0000","last_undegraded":"2024-04-15T15:15:15.865605+0000","last_fullsized":"2024-04-15T15:15:15.865605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865784+0000","last_change":"2024-04-15T15:15:10.298282+0000","last_active":"2024-04-15T15:15:15.865784+0000","last_peered":"2024-04-15T15:15:15.865784+0000","last_clean":"2024-04-15T15:15:15.865784+0000","last_became_active":"2024-04-15T15:15:10.297734+0000","last_became_peered":"2024-04-15T15:15:10.297734+0000","last_unstale":"2024-04-15T15:15:15.865784+0000","last_undegraded":"2024-04-15T15:15:15.865784+0000","last_fullsized":"2024-04-15T15:15:15.865784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309722+0000","last_change":"2024-04-15T15:15:14.321155+0000","last_active":"2024-04-15T15:15:15.309722+0000","last_peered":"2024-04-15T15:15:15.309722+0000","last_clean":"2024-04-15T15:15:15.309722+0000","last_became_active":"2024-04-15T15:15:14.320777+0000","last_became_peered":"2024-04-15T15:15:14.320777+0000","last_unstale":"2024-04-15T15:15:15.309722+0000","last_undegraded":"2024-04-15T15:15:15.309722+0000","last_fullsized":"2024-04-15T15:15:15.309722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:36.915564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310304+0000","last_change":"2024-04-15T15:15:10.288957+0000","last_active":"2024-04-15T15:15:15.310304+0000","last_peered":"2024-04-15T15:15:15.310304+0000","last_clean":"2024-04-15T15:15:15.310304+0000","last_became_active":"2024-04-15T15:15:10.288692+0000","last_became_peered":"2024-04-15T15:15:10.288692+0000","last_unstale":"2024-04-15T15:15:15.310304+0000","last_undegraded":"2024-04-15T15:15:15.310304+0000","last_fullsized":"2024-04-15T15:15:15.310304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865797+0000","last_change":"2024-04-15T15:15:09.295590+0000","last_active":"2024-04-15T15:15:15.865797+0000","last_peered":"2024-04-15T15:15:15.865797+0000","last_clean":"2024-04-15T15:15:15.865797+0000","last_became_active":"2024-04-15T15:15:09.295295+0000","last_became_peered":"2024-04-15T15:15:09.295295+0000","last_unstale":"2024-04-15T15:15:15.865797+0000","last_undegraded":"2024-04-15T15:15:15.865797+0000","last_fullsized":"2024-04-15T15:15:15.865797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309855+0000","last_change":"2024-04-15T15:15:14.327835+0000","last_active":"2024-04-15T15:15:15.309855+0000","last_peered":"2024-04-15T15:15:15.309855+0000","last_clean":"2024-04-15T15:15:15.309855+0000","last_became_active":"2024-04-15T15:15:14.327245+0000","last_became_peered":"2024-04-15T15:15:14.327245+0000","last_unstale":"2024-04-15T15:15:15.309855+0000","last_undegraded":"2024-04-15T15:15:15.309855+0000","last_fullsized":"2024-04-15T15:15:15.309855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308188+0000","last_change":"2024-04-15T15:15:10.295527+0000","last_active":"2024-04-15T15:15:15.308188+0000","last_peered":"2024-04-15T15:15:15.308188+0000","last_clean":"2024-04-15T15:15:15.308188+0000","last_became_active":"2024-04-15T15:15:10.295265+0000","last_became_peered":"2024-04-15T15:15:10.295265+0000","last_unstale":"2024-04-15T15:15:15.308188+0000","last_undegraded":"2024-04-15T15:15:15.308188+0000","last_fullsized":"2024-04-15T15:15:15.308188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393749+0000","last_change":"2024-04-15T15:15:09.291151+0000","last_active":"2024-04-15T15:15:15.393749+0000","last_peered":"2024-04-15T15:15:15.393749+0000","last_clean":"2024-04-15T15:15:15.393749+0000","last_became_active":"2024-04-15T15:15:09.290851+0000","last_became_peered":"2024-04-15T15:15:09.290851+0000","last_unstale":"2024-04-15T15:15:15.393749+0000","last_undegraded":"2024-04-15T15:15:15.393749+0000","last_fullsized":"2024-04-15T15:15:15.393749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309375+0000","last_change":"2024-04-15T15:15:14.327866+0000","last_active":"2024-04-15T15:15:15.309375+0000","last_peered":"2024-04-15T15:15:15.309375+0000","last_clean":"2024-04-15T15:15:15.309375+0000","last_became_active":"2024-04-15T15:15:14.327398+0000","last_became_peered":"2024-04-15T15:15:14.327398+0000","last_unstale":"2024-04-15T15:15:15.309375+0000","last_undegraded":"2024-04-15T15:15:15.309375+0000","last_fullsized":"2024-04-15T15:15:15.309375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:45.919286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308185+0000","last_change":"2024-04-15T15:15:10.294406+0000","last_active":"2024-04-15T15:15:15.308185+0000","last_peered":"2024-04-15T15:15:15.308185+0000","last_clean":"2024-04-15T15:15:15.308185+0000","last_became_active":"2024-04-15T15:15:10.294005+0000","last_became_peered":"2024-04-15T15:15:10.294005+0000","last_unstale":"2024-04-15T15:15:15.308185+0000","last_undegraded":"2024-04-15T15:15:15.308185+0000","last_fullsized":"2024-04-15T15:15:15.308185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393556+0000","last_change":"2024-04-15T15:15:09.288397+0000","last_active":"2024-04-15T15:15:15.393556+0000","last_peered":"2024-04-15T15:15:15.393556+0000","last_clean":"2024-04-15T15:15:15.393556+0000","last_became_active":"2024-04-15T15:15:09.288135+0000","last_became_peered":"2024-04-15T15:15:09.288135+0000","last_unstale":"2024-04-15T15:15:15.393556+0000","last_undegraded":"2024-04-15T15:15:15.393556+0000","last_fullsized":"2024-04-15T15:15:15.393556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394110+0000","last_change":"2024-04-15T15:15:14.488972+0000","last_active":"2024-04-15T15:15:15.394110+0000","last_peered":"2024-04-15T15:15:15.394110+0000","last_clean":"2024-04-15T15:15:15.394110+0000","last_became_active":"2024-04-15T15:15:14.488302+0000","last_became_peered":"2024-04-15T15:15:14.488302+0000","last_unstale":"2024-04-15T15:15:15.394110+0000","last_undegraded":"2024-04-15T15:15:15.394110+0000","last_fullsized":"2024-04-15T15:15:15.394110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310517+0000","last_change":"2024-04-15T15:15:10.294428+0000","last_active":"2024-04-15T15:15:15.310517+0000","last_peered":"2024-04-15T15:15:15.310517+0000","last_clean":"2024-04-15T15:15:15.310517+0000","last_became_active":"2024-04-15T15:15:10.294121+0000","last_became_peered":"2024-04-15T15:15:10.294121+0000","last_unstale":"2024-04-15T15:15:15.310517+0000","last_undegraded":"2024-04-15T15:15:15.310517+0000","last_fullsized":"2024-04-15T15:15:15.310517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:06.364679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865099+0000","last_change":"2024-04-15T15:15:09.294900+0000","last_active":"2024-04-15T15:15:15.865099+0000","last_peered":"2024-04-15T15:15:15.865099+0000","last_clean":"2024-04-15T15:15:15.865099+0000","last_became_active":"2024-04-15T15:15:09.294601+0000","last_became_peered":"2024-04-15T15:15:09.294601+0000","last_unstale":"2024-04-15T15:15:15.865099+0000","last_undegraded":"2024-04-15T15:15:15.865099+0000","last_fullsized":"2024-04-15T15:15:15.865099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309437+0000","last_change":"2024-04-15T15:15:14.328664+0000","last_active":"2024-04-15T15:15:15.309437+0000","last_peered":"2024-04-15T15:15:15.309437+0000","last_clean":"2024-04-15T15:15:15.309437+0000","last_became_active":"2024-04-15T15:15:14.327796+0000","last_became_peered":"2024-04-15T15:15:14.327796+0000","last_unstale":"2024-04-15T15:15:15.309437+0000","last_undegraded":"2024-04-15T15:15:15.309437+0000","last_fullsized":"2024-04-15T15:15:15.309437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:28.892025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865447+0000","last_change":"2024-04-15T15:15:10.288747+0000","last_active":"2024-04-15T15:15:15.865447+0000","last_peered":"2024-04-15T15:15:15.865447+0000","last_clean":"2024-04-15T15:15:15.865447+0000","last_became_active":"2024-04-15T15:15:10.288465+0000","last_became_peered":"2024-04-15T15:15:10.288465+0000","last_unstale":"2024-04-15T15:15:15.865447+0000","last_undegraded":"2024-04-15T15:15:15.865447+0000","last_fullsized":"2024-04-15T15:15:15.865447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311197+0000","last_change":"2024-04-15T15:15:09.291999+0000","last_active":"2024-04-15T15:15:15.311197+0000","last_peered":"2024-04-15T15:15:15.311197+0000","last_clean":"2024-04-15T15:15:15.311197+0000","last_became_active":"2024-04-15T15:15:09.291744+0000","last_became_peered":"2024-04-15T15:15:09.291744+0000","last_unstale":"2024-04-15T15:15:15.311197+0000","last_undegraded":"2024-04-15T15:15:15.311197+0000","last_fullsized":"2024-04-15T15:15:15.311197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310618+0000","last_change":"2024-04-15T15:15:14.491129+0000","last_active":"2024-04-15T15:15:15.310618+0000","last_peered":"2024-04-15T15:15:15.310618+0000","last_clean":"2024-04-15T15:15:15.310618+0000","last_became_active":"2024-04-15T15:15:14.490619+0000","last_became_peered":"2024-04-15T15:15:14.490619+0000","last_unstale":"2024-04-15T15:15:15.310618+0000","last_undegraded":"2024-04-15T15:15:15.310618+0000","last_fullsized":"2024-04-15T15:15:15.310618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:46.083837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308841+0000","last_change":"2024-04-15T15:15:10.297516+0000","last_active":"2024-04-15T15:15:15.308841+0000","last_peered":"2024-04-15T15:15:15.308841+0000","last_clean":"2024-04-15T15:15:15.308841+0000","last_became_active":"2024-04-15T15:15:10.297282+0000","last_became_peered":"2024-04-15T15:15:10.297282+0000","last_unstale":"2024-04-15T15:15:15.308841+0000","last_undegraded":"2024-04-15T15:15:15.308841+0000","last_fullsized":"2024-04-15T15:15:15.308841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394314+0000","last_change":"2024-04-15T15:15:09.288389+0000","last_active":"2024-04-15T15:15:15.394314+0000","last_peered":"2024-04-15T15:15:15.394314+0000","last_clean":"2024-04-15T15:15:15.394314+0000","last_became_active":"2024-04-15T15:15:09.288101+0000","last_became_peered":"2024-04-15T15:15:09.288101+0000","last_unstale":"2024-04-15T15:15:15.394314+0000","last_undegraded":"2024-04-15T15:15:15.394314+0000","last_fullsized":"2024-04-15T15:15:15.394314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.307926+0000","last_change":"2024-04-15T15:15:14.488661+0000","last_active":"2024-04-15T15:15:15.307926+0000","last_peered":"2024-04-15T15:15:15.307926+0000","last_clean":"2024-04-15T15:15:15.307926+0000","last_became_active":"2024-04-15T15:15:14.487944+0000","last_became_peered":"2024-04-15T15:15:14.487944+0000","last_unstale":"2024-04-15T15:15:15.307926+0000","last_undegraded":"2024-04-15T15:15:15.307926+0000","last_fullsized":"2024-04-15T15:15:15.307926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:36.475872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393929+0000","last_change":"2024-04-15T15:15:10.289924+0000","last_active":"2024-04-15T15:15:15.393929+0000","last_peered":"2024-04-15T15:15:15.393929+0000","last_clean":"2024-04-15T15:15:15.393929+0000","last_became_active":"2024-04-15T15:15:10.289662+0000","last_became_peered":"2024-04-15T15:15:10.289662+0000","last_unstale":"2024-04-15T15:15:15.393929+0000","last_undegraded":"2024-04-15T15:15:15.393929+0000","last_fullsized":"2024-04-15T15:15:15.393929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866240+0000","last_change":"2024-04-15T15:15:09.295866+0000","last_active":"2024-04-15T15:15:15.866240+0000","last_peered":"2024-04-15T15:15:15.866240+0000","last_clean":"2024-04-15T15:15:15.866240+0000","last_became_active":"2024-04-15T15:15:09.295507+0000","last_became_peered":"2024-04-15T15:15:09.295507+0000","last_unstale":"2024-04-15T15:15:15.866240+0000","last_undegraded":"2024-04-15T15:15:15.866240+0000","last_fullsized":"2024-04-15T15:15:15.866240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:01.918524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865520+0000","last_change":"2024-04-15T15:15:14.487233+0000","last_active":"2024-04-15T15:15:15.865520+0000","last_peered":"2024-04-15T15:15:15.865520+0000","last_clean":"2024-04-15T15:15:15.865520+0000","last_became_active":"2024-04-15T15:15:14.486846+0000","last_became_peered":"2024-04-15T15:15:14.486846+0000","last_unstale":"2024-04-15T15:15:15.865520+0000","last_undegraded":"2024-04-15T15:15:15.865520+0000","last_fullsized":"2024-04-15T15:15:15.865520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865899+0000","last_change":"2024-04-15T15:15:10.299154+0000","last_active":"2024-04-15T15:15:15.865899+0000","last_peered":"2024-04-15T15:15:15.865899+0000","last_clean":"2024-04-15T15:15:15.865899+0000","last_became_active":"2024-04-15T15:15:10.298711+0000","last_became_peered":"2024-04-15T15:15:10.298711+0000","last_unstale":"2024-04-15T15:15:15.865899+0000","last_undegraded":"2024-04-15T15:15:15.865899+0000","last_fullsized":"2024-04-15T15:15:15.865899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394042+0000","last_change":"2024-04-15T15:15:09.289347+0000","last_active":"2024-04-15T15:15:15.394042+0000","last_peered":"2024-04-15T15:15:15.394042+0000","last_clean":"2024-04-15T15:15:15.394042+0000","last_became_active":"2024-04-15T15:15:09.289062+0000","last_became_peered":"2024-04-15T15:15:09.289062+0000","last_unstale":"2024-04-15T15:15:15.394042+0000","last_undegraded":"2024-04-15T15:15:15.394042+0000","last_fullsized":"2024-04-15T15:15:15.394042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393588+0000","last_change":"2024-04-15T15:15:14.330539+0000","last_active":"2024-04-15T15:15:15.393588+0000","last_peered":"2024-04-15T15:15:15.393588+0000","last_clean":"2024-04-15T15:15:15.393588+0000","last_became_active":"2024-04-15T15:15:14.330173+0000","last_became_peered":"2024-04-15T15:15:14.330173+0000","last_unstale":"2024-04-15T15:15:15.393588+0000","last_undegraded":"2024-04-15T15:15:15.393588+0000","last_fullsized":"2024-04-15T15:15:15.393588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310032+0000","last_change":"2024-04-15T15:15:10.286603+0000","last_active":"2024-04-15T15:15:15.310032+0000","last_peered":"2024-04-15T15:15:15.310032+0000","last_clean":"2024-04-15T15:15:15.310032+0000","last_became_active":"2024-04-15T15:15:10.286306+0000","last_became_peered":"2024-04-15T15:15:10.286306+0000","last_unstale":"2024-04-15T15:15:15.310032+0000","last_undegraded":"2024-04-15T15:15:15.310032+0000","last_fullsized":"2024-04-15T15:15:15.310032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:47.785526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865601+0000","last_change":"2024-04-15T15:15:09.291918+0000","last_active":"2024-04-15T15:15:15.865601+0000","last_peered":"2024-04-15T15:15:15.865601+0000","last_clean":"2024-04-15T15:15:15.865601+0000","last_became_active":"2024-04-15T15:15:09.291664+0000","last_became_peered":"2024-04-15T15:15:09.291664+0000","last_unstale":"2024-04-15T15:15:15.865601+0000","last_undegraded":"2024-04-15T15:15:15.865601+0000","last_fullsized":"2024-04-15T15:15:15.865601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393983+0000","last_change":"2024-04-15T15:15:14.329421+0000","last_active":"2024-04-15T15:15:15.393983+0000","last_peered":"2024-04-15T15:15:15.393983+0000","last_clean":"2024-04-15T15:15:15.393983+0000","last_became_active":"2024-04-15T15:15:14.329120+0000","last_became_peered":"2024-04-15T15:15:14.329120+0000","last_unstale":"2024-04-15T15:15:15.393983+0000","last_undegraded":"2024-04-15T15:15:15.393983+0000","last_fullsized":"2024-04-15T15:15:15.393983+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393608+0000","last_change":"2024-04-15T15:15:10.290334+0000","last_active":"2024-04-15T15:15:15.393608+0000","last_peered":"2024-04-15T15:15:15.393608+0000","last_clean":"2024-04-15T15:15:15.393608+0000","last_became_active":"2024-04-15T15:15:10.290075+0000","last_became_peered":"2024-04-15T15:15:10.290075+0000","last_unstale":"2024-04-15T15:15:15.393608+0000","last_undegraded":"2024-04-15T15:15:15.393608+0000","last_fullsized":"2024-04-15T15:15:15.393608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864908+0000","last_change":"2024-04-15T15:15:09.291564+0000","last_active":"2024-04-15T15:15:15.864908+0000","last_peered":"2024-04-15T15:15:15.864908+0000","last_clean":"2024-04-15T15:15:15.864908+0000","last_became_active":"2024-04-15T15:15:09.291265+0000","last_became_peered":"2024-04-15T15:15:09.291265+0000","last_unstale":"2024-04-15T15:15:15.864908+0000","last_undegraded":"2024-04-15T15:15:15.864908+0000","last_fullsized":"2024-04-15T15:15:15.864908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394137+0000","last_change":"2024-04-15T15:15:14.329535+0000","last_active":"2024-04-15T15:15:15.394137+0000","last_peered":"2024-04-15T15:15:15.394137+0000","last_clean":"2024-04-15T15:15:15.394137+0000","last_became_active":"2024-04-15T15:15:14.329173+0000","last_became_peered":"2024-04-15T15:15:14.329173+0000","last_unstale":"2024-04-15T15:15:15.394137+0000","last_undegraded":"2024-04-15T15:15:15.394137+0000","last_fullsized":"2024-04-15T15:15:15.394137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864715+0000","last_change":"2024-04-15T15:15:10.292048+0000","last_active":"2024-04-15T15:15:15.864715+0000","last_peered":"2024-04-15T15:15:15.864715+0000","last_clean":"2024-04-15T15:15:15.864715+0000","last_became_active":"2024-04-15T15:15:10.291773+0000","last_became_peered":"2024-04-15T15:15:10.291773+0000","last_unstale":"2024-04-15T15:15:15.864715+0000","last_undegraded":"2024-04-15T15:15:15.864715+0000","last_fullsized":"2024-04-15T15:15:15.864715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:45.215354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865214+0000","last_change":"2024-04-15T15:15:09.295426+0000","last_active":"2024-04-15T15:15:15.865214+0000","last_peered":"2024-04-15T15:15:15.865214+0000","last_clean":"2024-04-15T15:15:15.865214+0000","last_became_active":"2024-04-15T15:15:09.295096+0000","last_became_peered":"2024-04-15T15:15:09.295096+0000","last_unstale":"2024-04-15T15:15:15.865214+0000","last_undegraded":"2024-04-15T15:15:15.865214+0000","last_fullsized":"2024-04-15T15:15:15.865214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865346+0000","last_change":"2024-04-15T15:15:14.487282+0000","last_active":"2024-04-15T15:15:15.865346+0000","last_peered":"2024-04-15T15:15:15.865346+0000","last_clean":"2024-04-15T15:15:15.865346+0000","last_became_active":"2024-04-15T15:15:14.486873+0000","last_became_peered":"2024-04-15T15:15:14.486873+0000","last_unstale":"2024-04-15T15:15:15.865346+0000","last_undegraded":"2024-04-15T15:15:15.865346+0000","last_fullsized":"2024-04-15T15:15:15.865346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308149+0000","last_change":"2024-04-15T15:15:10.301516+0000","last_active":"2024-04-15T15:15:15.308149+0000","last_peered":"2024-04-15T15:15:15.308149+0000","last_clean":"2024-04-15T15:15:15.308149+0000","last_became_active":"2024-04-15T15:15:10.301272+0000","last_became_peered":"2024-04-15T15:15:10.301272+0000","last_unstale":"2024-04-15T15:15:15.308149+0000","last_undegraded":"2024-04-15T15:15:15.308149+0000","last_fullsized":"2024-04-15T15:15:15.308149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865920+0000","last_change":"2024-04-15T15:15:09.292652+0000","last_active":"2024-04-15T15:15:15.865920+0000","last_peered":"2024-04-15T15:15:15.865920+0000","last_clean":"2024-04-15T15:15:15.865920+0000","last_became_active":"2024-04-15T15:15:09.292366+0000","last_became_peered":"2024-04-15T15:15:09.292366+0000","last_unstale":"2024-04-15T15:15:15.865920+0000","last_undegraded":"2024-04-15T15:15:15.865920+0000","last_fullsized":"2024-04-15T15:15:15.865920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865328+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:15:15.865328+0000","last_peered":"2024-04-15T15:15:15.865328+0000","last_clean":"2024-04-15T15:15:15.865328+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:15:15.865328+0000","last_undegraded":"2024-04-15T15:15:15.865328+0000","last_fullsized":"2024-04-15T15:15:15.865328+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310218+0000","last_change":"2024-04-15T15:15:14.488973+0000","last_active":"2024-04-15T15:15:15.310218+0000","last_peered":"2024-04-15T15:15:15.310218+0000","last_clean":"2024-04-15T15:15:15.310218+0000","last_became_active":"2024-04-15T15:15:14.488151+0000","last_became_peered":"2024-04-15T15:15:14.488151+0000","last_unstale":"2024-04-15T15:15:15.310218+0000","last_undegraded":"2024-04-15T15:15:15.310218+0000","last_fullsized":"2024-04-15T15:15:15.310218+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865992+0000","last_change":"2024-04-15T15:15:10.303487+0000","last_active":"2024-04-15T15:15:15.865992+0000","last_peered":"2024-04-15T15:15:15.865992+0000","last_clean":"2024-04-15T15:15:15.865992+0000","last_became_active":"2024-04-15T15:15:10.303222+0000","last_became_peered":"2024-04-15T15:15:10.303222+0000","last_unstale":"2024-04-15T15:15:15.865992+0000","last_undegraded":"2024-04-15T15:15:15.865992+0000","last_fullsized":"2024-04-15T15:15:15.865992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:09.292763+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:09.292482+0000","last_became_peered":"2024-04-15T15:15:09.292482+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310239+0000","last_change":"2024-04-15T15:15:14.487139+0000","last_active":"2024-04-15T15:15:15.310239+0000","last_peered":"2024-04-15T15:15:15.310239+0000","last_clean":"2024-04-15T15:15:15.310239+0000","last_became_active":"2024-04-15T15:15:14.486746+0000","last_became_peered":"2024-04-15T15:15:14.486746+0000","last_unstale":"2024-04-15T15:15:15.310239+0000","last_undegraded":"2024-04-15T15:15:15.310239+0000","last_fullsized":"2024-04-15T15:15:15.310239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865734+0000","last_change":"2024-04-15T15:15:10.298528+0000","last_active":"2024-04-15T15:15:15.865734+0000","last_peered":"2024-04-15T15:15:15.865734+0000","last_clean":"2024-04-15T15:15:15.865734+0000","last_became_active":"2024-04-15T15:15:10.298175+0000","last_became_peered":"2024-04-15T15:15:10.298175+0000","last_unstale":"2024-04-15T15:15:15.865734+0000","last_undegraded":"2024-04-15T15:15:15.865734+0000","last_fullsized":"2024-04-15T15:15:15.865734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394078+0000","last_change":"2024-04-15T15:15:09.287274+0000","last_active":"2024-04-15T15:15:15.394078+0000","last_peered":"2024-04-15T15:15:15.394078+0000","last_clean":"2024-04-15T15:15:15.394078+0000","last_became_active":"2024-04-15T15:15:09.287022+0000","last_became_peered":"2024-04-15T15:15:09.287022+0000","last_unstale":"2024-04-15T15:15:15.394078+0000","last_undegraded":"2024-04-15T15:15:15.394078+0000","last_fullsized":"2024-04-15T15:15:15.394078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865617+0000","last_change":"2024-04-15T15:15:14.327367+0000","last_active":"2024-04-15T15:15:15.865617+0000","last_peered":"2024-04-15T15:15:15.865617+0000","last_clean":"2024-04-15T15:15:15.865617+0000","last_became_active":"2024-04-15T15:15:14.327012+0000","last_became_peered":"2024-04-15T15:15:14.327012+0000","last_unstale":"2024-04-15T15:15:15.865617+0000","last_undegraded":"2024-04-15T15:15:15.865617+0000","last_fullsized":"2024-04-15T15:15:15.865617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866362+0000","last_change":"2024-04-15T15:15:10.300664+0000","last_active":"2024-04-15T15:15:15.866362+0000","last_peered":"2024-04-15T15:15:15.866362+0000","last_clean":"2024-04-15T15:15:15.866362+0000","last_became_active":"2024-04-15T15:15:10.300411+0000","last_became_peered":"2024-04-15T15:15:10.300411+0000","last_unstale":"2024-04-15T15:15:15.866362+0000","last_undegraded":"2024-04-15T15:15:15.866362+0000","last_fullsized":"2024-04-15T15:15:15.866362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:32.450402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864760+0000","last_change":"2024-04-15T15:15:09.291588+0000","last_active":"2024-04-15T15:15:15.864760+0000","last_peered":"2024-04-15T15:15:15.864760+0000","last_clean":"2024-04-15T15:15:15.864760+0000","last_became_active":"2024-04-15T15:15:09.291317+0000","last_became_peered":"2024-04-15T15:15:09.291317+0000","last_unstale":"2024-04-15T15:15:15.864760+0000","last_undegraded":"2024-04-15T15:15:15.864760+0000","last_fullsized":"2024-04-15T15:15:15.864760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:14.488073+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:14.487630+0000","last_became_peered":"2024-04-15T15:15:14.487630+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866072+0000","last_change":"2024-04-15T15:15:10.293789+0000","last_active":"2024-04-15T15:15:15.866072+0000","last_peered":"2024-04-15T15:15:15.866072+0000","last_clean":"2024-04-15T15:15:15.866072+0000","last_became_active":"2024-04-15T15:15:10.293508+0000","last_became_peered":"2024-04-15T15:15:10.293508+0000","last_unstale":"2024-04-15T15:15:15.866072+0000","last_undegraded":"2024-04-15T15:15:15.866072+0000","last_fullsized":"2024-04-15T15:15:15.866072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:13.731730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310273+0000","last_change":"2024-04-15T15:15:09.280627+0000","last_active":"2024-04-15T15:15:15.310273+0000","last_peered":"2024-04-15T15:15:15.310273+0000","last_clean":"2024-04-15T15:15:15.310273+0000","last_became_active":"2024-04-15T15:15:09.280327+0000","last_became_peered":"2024-04-15T15:15:09.280327+0000","last_unstale":"2024-04-15T15:15:15.310273+0000","last_undegraded":"2024-04-15T15:15:15.310273+0000","last_fullsized":"2024-04-15T15:15:15.310273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:24.741048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308985+0000","last_change":"2024-04-15T15:15:14.327503+0000","last_active":"2024-04-15T15:15:15.308985+0000","last_peered":"2024-04-15T15:15:15.308985+0000","last_clean":"2024-04-15T15:15:15.308985+0000","last_became_active":"2024-04-15T15:15:14.327045+0000","last_became_peered":"2024-04-15T15:15:14.327045+0000","last_unstale":"2024-04-15T15:15:15.308985+0000","last_undegraded":"2024-04-15T15:15:15.308985+0000","last_fullsized":"2024-04-15T15:15:15.308985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:41.707546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865253+0000","last_change":"2024-04-15T15:15:10.294511+0000","last_active":"2024-04-15T15:15:15.865253+0000","last_peered":"2024-04-15T15:15:15.865253+0000","last_clean":"2024-04-15T15:15:15.865253+0000","last_became_active":"2024-04-15T15:15:10.294217+0000","last_became_peered":"2024-04-15T15:15:10.294217+0000","last_unstale":"2024-04-15T15:15:15.865253+0000","last_undegraded":"2024-04-15T15:15:15.865253+0000","last_fullsized":"2024-04-15T15:15:15.865253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864350+0000","last_change":"2024-04-15T15:15:09.289902+0000","last_active":"2024-04-15T15:15:15.864350+0000","last_peered":"2024-04-15T15:15:15.864350+0000","last_clean":"2024-04-15T15:15:15.864350+0000","last_became_active":"2024-04-15T15:15:09.289521+0000","last_became_peered":"2024-04-15T15:15:09.289521+0000","last_unstale":"2024-04-15T15:15:15.864350+0000","last_undegraded":"2024-04-15T15:15:15.864350+0000","last_fullsized":"2024-04-15T15:15:15.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865201+0000","last_change":"2024-04-15T15:15:14.325169+0000","last_active":"2024-04-15T15:15:15.865201+0000","last_peered":"2024-04-15T15:15:15.865201+0000","last_clean":"2024-04-15T15:15:15.865201+0000","last_became_active":"2024-04-15T15:15:14.324680+0000","last_became_peered":"2024-04-15T15:15:14.324680+0000","last_unstale":"2024-04-15T15:15:15.865201+0000","last_undegraded":"2024-04-15T15:15:15.865201+0000","last_fullsized":"2024-04-15T15:15:15.865201+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308937+0000","last_change":"2024-04-15T15:15:10.291452+0000","last_active":"2024-04-15T15:15:15.308937+0000","last_peered":"2024-04-15T15:15:15.308937+0000","last_clean":"2024-04-15T15:15:15.308937+0000","last_became_active":"2024-04-15T15:15:10.291170+0000","last_became_peered":"2024-04-15T15:15:10.291170+0000","last_unstale":"2024-04-15T15:15:15.308937+0000","last_undegraded":"2024-04-15T15:15:15.308937+0000","last_fullsized":"2024-04-15T15:15:15.308937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309951+0000","last_change":"2024-04-15T15:15:09.292208+0000","last_active":"2024-04-15T15:15:15.309951+0000","last_peered":"2024-04-15T15:15:15.309951+0000","last_clean":"2024-04-15T15:15:15.309951+0000","last_became_active":"2024-04-15T15:15:09.290702+0000","last_became_peered":"2024-04-15T15:15:09.290702+0000","last_unstale":"2024-04-15T15:15:15.309951+0000","last_undegraded":"2024-04-15T15:15:15.309951+0000","last_fullsized":"2024-04-15T15:15:15.309951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308793+0000","last_change":"2024-04-15T15:15:14.488657+0000","last_active":"2024-04-15T15:15:15.308793+0000","last_peered":"2024-04-15T15:15:15.308793+0000","last_clean":"2024-04-15T15:15:15.308793+0000","last_became_active":"2024-04-15T15:15:14.488282+0000","last_became_peered":"2024-04-15T15:15:14.488282+0000","last_unstale":"2024-04-15T15:15:15.308793+0000","last_undegraded":"2024-04-15T15:15:15.308793+0000","last_fullsized":"2024-04-15T15:15:15.308793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:21:03.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865696+0000","last_change":"2024-04-15T15:15:10.301823+0000","last_active":"2024-04-15T15:15:15.865696+0000","last_peered":"2024-04-15T15:15:15.865696+0000","last_clean":"2024-04-15T15:15:15.865696+0000","last_became_active":"2024-04-15T15:15:10.301545+0000","last_became_peered":"2024-04-15T15:15:10.301545+0000","last_unstale":"2024-04-15T15:15:15.865696+0000","last_undegraded":"2024-04-15T15:15:15.865696+0000","last_fullsized":"2024-04-15T15:15:15.865696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865481+0000","last_change":"2024-04-15T15:15:09.291724+0000","last_active":"2024-04-15T15:15:15.865481+0000","last_peered":"2024-04-15T15:15:15.865481+0000","last_clean":"2024-04-15T15:15:15.865481+0000","last_became_active":"2024-04-15T15:15:09.291390+0000","last_became_peered":"2024-04-15T15:15:09.291390+0000","last_unstale":"2024-04-15T15:15:15.865481+0000","last_undegraded":"2024-04-15T15:15:15.865481+0000","last_fullsized":"2024-04-15T15:15:15.865481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308524+0000","last_change":"2024-04-15T15:15:14.491287+0000","last_active":"2024-04-15T15:15:15.308524+0000","last_peered":"2024-04-15T15:15:15.308524+0000","last_clean":"2024-04-15T15:15:15.308524+0000","last_became_active":"2024-04-15T15:15:14.488643+0000","last_became_peered":"2024-04-15T15:15:14.488643+0000","last_unstale":"2024-04-15T15:15:15.308524+0000","last_undegraded":"2024-04-15T15:15:15.308524+0000","last_fullsized":"2024-04-15T15:15:15.308524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:29:47.252030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865184+0000","last_change":"2024-04-15T15:15:10.293687+0000","last_active":"2024-04-15T15:15:15.865184+0000","last_peered":"2024-04-15T15:15:15.865184+0000","last_clean":"2024-04-15T15:15:15.865184+0000","last_became_active":"2024-04-15T15:15:10.293430+0000","last_became_peered":"2024-04-15T15:15:10.293430+0000","last_unstale":"2024-04-15T15:15:15.865184+0000","last_undegraded":"2024-04-15T15:15:15.865184+0000","last_fullsized":"2024-04-15T15:15:15.865184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865653+0000","last_change":"2024-04-15T15:15:09.282027+0000","last_active":"2024-04-15T15:15:15.865653+0000","last_peered":"2024-04-15T15:15:15.865653+0000","last_clean":"2024-04-15T15:15:15.865653+0000","last_became_active":"2024-04-15T15:15:09.281730+0000","last_became_peered":"2024-04-15T15:15:09.281730+0000","last_unstale":"2024-04-15T15:15:15.865653+0000","last_undegraded":"2024-04-15T15:15:15.865653+0000","last_fullsized":"2024-04-15T15:15:15.865653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:01.954710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866059+0000","last_change":"2024-04-15T15:15:14.489189+0000","last_active":"2024-04-15T15:15:15.866059+0000","last_peered":"2024-04-15T15:15:15.866059+0000","last_clean":"2024-04-15T15:15:15.866059+0000","last_became_active":"2024-04-15T15:15:14.488758+0000","last_became_peered":"2024-04-15T15:15:14.488758+0000","last_unstale":"2024-04-15T15:15:15.866059+0000","last_undegraded":"2024-04-15T15:15:15.866059+0000","last_fullsized":"2024-04-15T15:15:15.866059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865898+0000","last_change":"2024-04-15T15:15:10.298047+0000","last_active":"2024-04-15T15:15:15.865898+0000","last_peered":"2024-04-15T15:15:15.865898+0000","last_clean":"2024-04-15T15:15:15.865898+0000","last_became_active":"2024-04-15T15:15:10.297779+0000","last_became_peered":"2024-04-15T15:15:10.297779+0000","last_unstale":"2024-04-15T15:15:15.865898+0000","last_undegraded":"2024-04-15T15:15:15.865898+0000","last_fullsized":"2024-04-15T15:15:15.865898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:19.312104+0000","last_change":"2024-04-15T15:15:09.290413+0000","last_active":"2024-04-15T15:15:19.312104+0000","last_peered":"2024-04-15T15:15:19.312104+0000","last_clean":"2024-04-15T15:15:19.312104+0000","last_became_active":"2024-04-15T15:15:09.289971+0000","last_became_peered":"2024-04-15T15:15:09.289971+0000","last_unstale":"2024-04-15T15:15:19.312104+0000","last_undegraded":"2024-04-15T15:15:19.312104+0000","last_fullsized":"2024-04-15T15:15:19.312104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393701+0000","last_change":"2024-04-15T15:15:14.324351+0000","last_active":"2024-04-15T15:15:15.393701+0000","last_peered":"2024-04-15T15:15:15.393701+0000","last_clean":"2024-04-15T15:15:15.393701+0000","last_became_active":"2024-04-15T15:15:14.322177+0000","last_became_peered":"2024-04-15T15:15:14.322177+0000","last_unstale":"2024-04-15T15:15:15.393701+0000","last_undegraded":"2024-04-15T15:15:15.393701+0000","last_fullsized":"2024-04-15T15:15:15.393701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310092+0000","last_change":"2024-04-15T15:15:10.290888+0000","last_active":"2024-04-15T15:15:15.310092+0000","last_peered":"2024-04-15T15:15:15.310092+0000","last_clean":"2024-04-15T15:15:15.310092+0000","last_became_active":"2024-04-15T15:15:10.290630+0000","last_became_peered":"2024-04-15T15:15:10.290630+0000","last_unstale":"2024-04-15T15:15:15.310092+0000","last_undegraded":"2024-04-15T15:15:15.310092+0000","last_fullsized":"2024-04-15T15:15:15.310092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394037+0000","last_change":"2024-04-15T15:15:09.291950+0000","last_active":"2024-04-15T15:15:15.394037+0000","last_peered":"2024-04-15T15:15:15.394037+0000","last_clean":"2024-04-15T15:15:15.394037+0000","last_became_active":"2024-04-15T15:15:09.291687+0000","last_became_peered":"2024-04-15T15:15:09.291687+0000","last_unstale":"2024-04-15T15:15:15.394037+0000","last_undegraded":"2024-04-15T15:15:15.394037+0000","last_fullsized":"2024-04-15T15:15:15.394037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309744+0000","last_change":"2024-04-15T15:15:14.328229+0000","last_active":"2024-04-15T15:15:15.309744+0000","last_peered":"2024-04-15T15:15:15.309744+0000","last_clean":"2024-04-15T15:15:15.309744+0000","last_became_active":"2024-04-15T15:15:14.327167+0000","last_became_peered":"2024-04-15T15:15:14.327167+0000","last_unstale":"2024-04-15T15:15:15.309744+0000","last_undegraded":"2024-04-15T15:15:15.309744+0000","last_fullsized":"2024-04-15T15:15:15.309744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:36.054042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865433+0000","last_change":"2024-04-15T15:15:10.300821+0000","last_active":"2024-04-15T15:15:15.865433+0000","last_peered":"2024-04-15T15:15:15.865433+0000","last_clean":"2024-04-15T15:15:15.865433+0000","last_became_active":"2024-04-15T15:15:10.300561+0000","last_became_peered":"2024-04-15T15:15:10.300561+0000","last_unstale":"2024-04-15T15:15:15.865433+0000","last_undegraded":"2024-04-15T15:15:15.865433+0000","last_fullsized":"2024-04-15T15:15:15.865433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865315+0000","last_change":"2024-04-15T15:15:09.294709+0000","last_active":"2024-04-15T15:15:15.865315+0000","last_peered":"2024-04-15T15:15:15.865315+0000","last_clean":"2024-04-15T15:15:15.865315+0000","last_became_active":"2024-04-15T15:15:09.294135+0000","last_became_peered":"2024-04-15T15:15:09.294135+0000","last_unstale":"2024-04-15T15:15:15.865315+0000","last_undegraded":"2024-04-15T15:15:15.865315+0000","last_fullsized":"2024-04-15T15:15:15.865315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308127+0000","last_change":"2024-04-15T15:15:14.332302+0000","last_active":"2024-04-15T15:15:15.308127+0000","last_peered":"2024-04-15T15:15:15.308127+0000","last_clean":"2024-04-15T15:15:15.308127+0000","last_became_active":"2024-04-15T15:15:14.331893+0000","last_became_peered":"2024-04-15T15:15:14.331893+0000","last_unstale":"2024-04-15T15:15:15.308127+0000","last_undegraded":"2024-04-15T15:15:15.308127+0000","last_fullsized":"2024-04-15T15:15:15.308127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:18:57.970154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865270+0000","last_change":"2024-04-15T15:15:10.295887+0000","last_active":"2024-04-15T15:15:15.865270+0000","last_peered":"2024-04-15T15:15:15.865270+0000","last_clean":"2024-04-15T15:15:15.865270+0000","last_became_active":"2024-04-15T15:15:10.295591+0000","last_became_peered":"2024-04-15T15:15:10.295591+0000","last_unstale":"2024-04-15T15:15:15.865270+0000","last_undegraded":"2024-04-15T15:15:15.865270+0000","last_fullsized":"2024-04-15T15:15:15.865270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311117+0000","last_change":"2024-04-15T15:15:09.290489+0000","last_active":"2024-04-15T15:15:15.311117+0000","last_peered":"2024-04-15T15:15:15.311117+0000","last_clean":"2024-04-15T15:15:15.311117+0000","last_became_active":"2024-04-15T15:15:09.290207+0000","last_became_peered":"2024-04-15T15:15:09.290207+0000","last_unstale":"2024-04-15T15:15:15.311117+0000","last_undegraded":"2024-04-15T15:15:15.311117+0000","last_fullsized":"2024-04-15T15:15:15.311117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308129+0000","last_change":"2024-04-15T15:15:14.490596+0000","last_active":"2024-04-15T15:15:15.308129+0000","last_peered":"2024-04-15T15:15:15.308129+0000","last_clean":"2024-04-15T15:15:15.308129+0000","last_became_active":"2024-04-15T15:15:14.489609+0000","last_became_peered":"2024-04-15T15:15:14.489609+0000","last_unstale":"2024-04-15T15:15:15.308129+0000","last_undegraded":"2024-04-15T15:15:15.308129+0000","last_fullsized":"2024-04-15T15:15:15.308129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865063+0000","last_change":"2024-04-15T15:15:14.489631+0000","last_active":"2024-04-15T15:15:15.865063+0000","last_peered":"2024-04-15T15:15:15.865063+0000","last_clean":"2024-04-15T15:15:15.865063+0000","last_became_active":"2024-04-15T15:15:14.489248+0000","last_became_peered":"2024-04-15T15:15:14.489248+0000","last_unstale":"2024-04-15T15:15:15.865063+0000","last_undegraded":"2024-04-15T15:15:15.865063+0000","last_fullsized":"2024-04-15T15:15:15.865063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393445+0000","last_change":"2024-04-15T15:15:10.293423+0000","last_active":"2024-04-15T15:15:15.393445+0000","last_peered":"2024-04-15T15:15:15.393445+0000","last_clean":"2024-04-15T15:15:15.393445+0000","last_became_active":"2024-04-15T15:15:10.293101+0000","last_became_peered":"2024-04-15T15:15:10.293101+0000","last_unstale":"2024-04-15T15:15:15.393445+0000","last_undegraded":"2024-04-15T15:15:15.393445+0000","last_fullsized":"2024-04-15T15:15:15.393445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865159+0000","last_change":"2024-04-15T15:15:09.290248+0000","last_active":"2024-04-15T15:15:15.865159+0000","last_peered":"2024-04-15T15:15:15.865159+0000","last_clean":"2024-04-15T15:15:15.865159+0000","last_became_active":"2024-04-15T15:15:09.289792+0000","last_became_peered":"2024-04-15T15:15:09.289792+0000","last_unstale":"2024-04-15T15:15:15.865159+0000","last_undegraded":"2024-04-15T15:15:15.865159+0000","last_fullsized":"2024-04-15T15:15:15.865159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311080+0000","last_change":"2024-04-15T15:15:14.323489+0000","last_active":"2024-04-15T15:15:15.311080+0000","last_peered":"2024-04-15T15:15:15.311080+0000","last_clean":"2024-04-15T15:15:15.311080+0000","last_became_active":"2024-04-15T15:15:14.322956+0000","last_became_peered":"2024-04-15T15:15:14.322956+0000","last_unstale":"2024-04-15T15:15:15.311080+0000","last_undegraded":"2024-04-15T15:15:15.311080+0000","last_fullsized":"2024-04-15T15:15:15.311080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308115+0000","last_change":"2024-04-15T15:15:10.295446+0000","last_active":"2024-04-15T15:15:15.308115+0000","last_peered":"2024-04-15T15:15:15.308115+0000","last_clean":"2024-04-15T15:15:15.308115+0000","last_became_active":"2024-04-15T15:15:10.295165+0000","last_became_peered":"2024-04-15T15:15:10.295165+0000","last_unstale":"2024-04-15T15:15:15.308115+0000","last_undegraded":"2024-04-15T15:15:15.308115+0000","last_fullsized":"2024-04-15T15:15:15.308115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308300+0000","last_change":"2024-04-15T15:15:09.289277+0000","last_active":"2024-04-15T15:15:15.308300+0000","last_peered":"2024-04-15T15:15:15.308300+0000","last_clean":"2024-04-15T15:15:15.308300+0000","last_became_active":"2024-04-15T15:15:09.289026+0000","last_became_peered":"2024-04-15T15:15:09.289026+0000","last_unstale":"2024-04-15T15:15:15.308300+0000","last_undegraded":"2024-04-15T15:15:15.308300+0000","last_fullsized":"2024-04-15T15:15:15.308300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310503+0000","last_change":"2024-04-15T15:15:09.285504+0000","last_active":"2024-04-15T15:15:15.310503+0000","last_peered":"2024-04-15T15:15:15.310503+0000","last_clean":"2024-04-15T15:15:15.310503+0000","last_became_active":"2024-04-15T15:15:09.285219+0000","last_became_peered":"2024-04-15T15:15:09.285219+0000","last_unstale":"2024-04-15T15:15:15.310503+0000","last_undegraded":"2024-04-15T15:15:15.310503+0000","last_fullsized":"2024-04-15T15:15:15.310503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865651+0000","last_change":"2024-04-15T15:15:10.297720+0000","last_active":"2024-04-15T15:15:15.865651+0000","last_peered":"2024-04-15T15:15:15.865651+0000","last_clean":"2024-04-15T15:15:15.865651+0000","last_became_active":"2024-04-15T15:15:10.297469+0000","last_became_peered":"2024-04-15T15:15:10.297469+0000","last_unstale":"2024-04-15T15:15:15.865651+0000","last_undegraded":"2024-04-15T15:15:15.865651+0000","last_fullsized":"2024-04-15T15:15:15.865651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864649+0000","last_change":"2024-04-15T15:15:14.327342+0000","last_active":"2024-04-15T15:15:15.864649+0000","last_peered":"2024-04-15T15:15:15.864649+0000","last_clean":"2024-04-15T15:15:15.864649+0000","last_became_active":"2024-04-15T15:15:14.326929+0000","last_became_peered":"2024-04-15T15:15:14.326929+0000","last_unstale":"2024-04-15T15:15:15.864649+0000","last_undegraded":"2024-04-15T15:15:15.864649+0000","last_fullsized":"2024-04-15T15:15:15.864649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:19:56.492223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308298+0000","last_change":"2024-04-15T15:15:09.288401+0000","last_active":"2024-04-15T15:15:15.308298+0000","last_peered":"2024-04-15T15:15:15.308298+0000","last_clean":"2024-04-15T15:15:15.308298+0000","last_became_active":"2024-04-15T15:15:09.288124+0000","last_became_peered":"2024-04-15T15:15:09.288124+0000","last_unstale":"2024-04-15T15:15:15.308298+0000","last_undegraded":"2024-04-15T15:15:15.308298+0000","last_fullsized":"2024-04-15T15:15:15.308298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309493+0000","last_change":"2024-04-15T15:15:10.293787+0000","last_active":"2024-04-15T15:15:15.309493+0000","last_peered":"2024-04-15T15:15:15.309493+0000","last_clean":"2024-04-15T15:15:15.309493+0000","last_became_active":"2024-04-15T15:15:10.293409+0000","last_became_peered":"2024-04-15T15:15:10.293409+0000","last_unstale":"2024-04-15T15:15:15.309493+0000","last_undegraded":"2024-04-15T15:15:15.309493+0000","last_fullsized":"2024-04-15T15:15:15.309493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:30.764771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308821+0000","last_change":"2024-04-15T15:15:14.323853+0000","last_active":"2024-04-15T15:15:15.308821+0000","last_peered":"2024-04-15T15:15:15.308821+0000","last_clean":"2024-04-15T15:15:15.308821+0000","last_became_active":"2024-04-15T15:15:14.323547+0000","last_became_peered":"2024-04-15T15:15:14.323547+0000","last_unstale":"2024-04-15T15:15:15.308821+0000","last_undegraded":"2024-04-15T15:15:15.308821+0000","last_fullsized":"2024-04-15T15:15:15.308821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:48.237751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308815+0000","last_change":"2024-04-15T15:15:10.290641+0000","last_active":"2024-04-15T15:15:15.308815+0000","last_peered":"2024-04-15T15:15:15.308815+0000","last_clean":"2024-04-15T15:15:15.308815+0000","last_became_active":"2024-04-15T15:15:10.290342+0000","last_became_peered":"2024-04-15T15:15:10.290342+0000","last_unstale":"2024-04-15T15:15:15.308815+0000","last_undegraded":"2024-04-15T15:15:15.308815+0000","last_fullsized":"2024-04-15T15:15:15.308815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864807+0000","last_change":"2024-04-15T15:15:10.302121+0000","last_active":"2024-04-15T15:15:15.864807+0000","last_peered":"2024-04-15T15:15:15.864807+0000","last_clean":"2024-04-15T15:15:15.864807+0000","last_became_active":"2024-04-15T15:15:10.301860+0000","last_became_peered":"2024-04-15T15:15:10.301860+0000","last_unstale":"2024-04-15T15:15:15.864807+0000","last_undegraded":"2024-04-15T15:15:15.864807+0000","last_fullsized":"2024-04-15T15:15:15.864807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865207+0000","last_change":"2024-04-15T15:15:10.298052+0000","last_active":"2024-04-15T15:15:15.865207+0000","last_peered":"2024-04-15T15:15:15.865207+0000","last_clean":"2024-04-15T15:15:15.865207+0000","last_became_active":"2024-04-15T15:15:10.296656+0000","last_became_peered":"2024-04-15T15:15:10.296656+0000","last_unstale":"2024-04-15T15:15:15.865207+0000","last_undegraded":"2024-04-15T15:15:15.865207+0000","last_fullsized":"2024-04-15T15:15:15.865207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865604+0000","last_change":"2024-04-15T15:15:10.296027+0000","last_active":"2024-04-15T15:15:15.865604+0000","last_peered":"2024-04-15T15:15:15.865604+0000","last_clean":"2024-04-15T15:15:15.865604+0000","last_became_active":"2024-04-15T15:15:10.295741+0000","last_became_peered":"2024-04-15T15:15:10.295741+0000","last_unstale":"2024-04-15T15:15:15.865604+0000","last_undegraded":"2024-04-15T15:15:15.865604+0000","last_fullsized":"2024-04-15T15:15:15.865604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308554+0000","last_change":"2024-04-15T15:15:10.287343+0000","last_active":"2024-04-15T15:15:15.308554+0000","last_peered":"2024-04-15T15:15:15.308554+0000","last_clean":"2024-04-15T15:15:15.308554+0000","last_became_active":"2024-04-15T15:15:10.287082+0000","last_became_peered":"2024-04-15T15:15:10.287082+0000","last_unstale":"2024-04-15T15:15:15.308554+0000","last_undegraded":"2024-04-15T15:15:15.308554+0000","last_fullsized":"2024-04-15T15:15:15.308554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393974+0000","last_change":"2024-04-15T15:15:10.289020+0000","last_active":"2024-04-15T15:15:15.393974+0000","last_peered":"2024-04-15T15:15:15.393974+0000","last_clean":"2024-04-15T15:15:15.393974+0000","last_became_active":"2024-04-15T15:15:10.288777+0000","last_became_peered":"2024-04-15T15:15:10.288777+0000","last_unstale":"2024-04-15T15:15:15.393974+0000","last_undegraded":"2024-04-15T15:15:15.393974+0000","last_fullsized":"2024-04-15T15:15:15.393974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310906+0000","last_change":"2024-04-15T15:15:10.290594+0000","last_active":"2024-04-15T15:15:15.310906+0000","last_peered":"2024-04-15T15:15:15.310906+0000","last_clean":"2024-04-15T15:15:15.310906+0000","last_became_active":"2024-04-15T15:15:10.290345+0000","last_became_peered":"2024-04-15T15:15:10.290345+0000","last_unstale":"2024-04-15T15:15:15.310906+0000","last_undegraded":"2024-04-15T15:15:15.310906+0000","last_fullsized":"2024-04-15T15:15:15.310906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308747+0000","last_change":"2024-04-15T15:15:10.297500+0000","last_active":"2024-04-15T15:15:15.308747+0000","last_peered":"2024-04-15T15:15:15.308747+0000","last_clean":"2024-04-15T15:15:15.308747+0000","last_became_active":"2024-04-15T15:15:10.296730+0000","last_became_peered":"2024-04-15T15:15:10.296730+0000","last_unstale":"2024-04-15T15:15:15.308747+0000","last_undegraded":"2024-04-15T15:15:15.308747+0000","last_fullsized":"2024-04-15T15:15:15.308747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865052+0000","last_change":"2024-04-15T15:15:10.296225+0000","last_active":"2024-04-15T15:15:15.865052+0000","last_peered":"2024-04-15T15:15:15.865052+0000","last_clean":"2024-04-15T15:15:15.865052+0000","last_became_active":"2024-04-15T15:15:10.295930+0000","last_became_peered":"2024-04-15T15:15:10.295930+0000","last_unstale":"2024-04-15T15:15:15.865052+0000","last_undegraded":"2024-04-15T15:15:15.865052+0000","last_fullsized":"2024-04-15T15:15:15.865052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308196+0000","last_change":"2024-04-15T15:15:10.289427+0000","last_active":"2024-04-15T15:15:15.308196+0000","last_peered":"2024-04-15T15:15:15.308196+0000","last_clean":"2024-04-15T15:15:15.308196+0000","last_became_active":"2024-04-15T15:15:10.289170+0000","last_became_peered":"2024-04-15T15:15:10.289170+0000","last_unstale":"2024-04-15T15:15:15.308196+0000","last_undegraded":"2024-04-15T15:15:15.308196+0000","last_fullsized":"2024-04-15T15:15:15.308196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865103+0000","last_change":"2024-04-15T15:15:10.304028+0000","last_active":"2024-04-15T15:15:15.865103+0000","last_peered":"2024-04-15T15:15:15.865103+0000","last_clean":"2024-04-15T15:15:15.865103+0000","last_became_active":"2024-04-15T15:15:10.303771+0000","last_became_peered":"2024-04-15T15:15:10.303771+0000","last_unstale":"2024-04-15T15:15:15.865103+0000","last_undegraded":"2024-04-15T15:15:15.865103+0000","last_fullsized":"2024-04-15T15:15:15.865103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865533+0000","last_change":"2024-04-15T15:15:10.293639+0000","last_active":"2024-04-15T15:15:15.865533+0000","last_peered":"2024-04-15T15:15:15.865533+0000","last_clean":"2024-04-15T15:15:15.865533+0000","last_became_active":"2024-04-15T15:15:10.293220+0000","last_became_peered":"2024-04-15T15:15:10.293220+0000","last_unstale":"2024-04-15T15:15:15.865533+0000","last_undegraded":"2024-04-15T15:15:15.865533+0000","last_fullsized":"2024-04-15T15:15:15.865533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310554+0000","last_change":"2024-04-15T15:15:10.289725+0000","last_active":"2024-04-15T15:15:15.310554+0000","last_peered":"2024-04-15T15:15:15.310554+0000","last_clean":"2024-04-15T15:15:15.310554+0000","last_became_active":"2024-04-15T15:15:10.289378+0000","last_became_peered":"2024-04-15T15:15:10.289378+0000","last_unstale":"2024-04-15T15:15:15.310554+0000","last_undegraded":"2024-04-15T15:15:15.310554+0000","last_fullsized":"2024-04-15T15:15:15.310554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866173+0000","last_change":"2024-04-15T15:15:10.295231+0000","last_active":"2024-04-15T15:15:15.866173+0000","last_peered":"2024-04-15T15:15:15.866173+0000","last_clean":"2024-04-15T15:15:15.866173+0000","last_became_active":"2024-04-15T15:15:10.294974+0000","last_became_peered":"2024-04-15T15:15:10.294974+0000","last_unstale":"2024-04-15T15:15:15.866173+0000","last_undegraded":"2024-04-15T15:15:15.866173+0000","last_fullsized":"2024-04-15T15:15:15.866173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864735+0000","last_change":"2024-04-15T15:15:10.300143+0000","last_active":"2024-04-15T15:15:15.864735+0000","last_peered":"2024-04-15T15:15:15.864735+0000","last_clean":"2024-04-15T15:15:15.864735+0000","last_became_active":"2024-04-15T15:15:10.299889+0000","last_became_peered":"2024-04-15T15:15:10.299889+0000","last_unstale":"2024-04-15T15:15:15.864735+0000","last_undegraded":"2024-04-15T15:15:15.864735+0000","last_fullsized":"2024-04-15T15:15:15.864735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:46.735175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865763+0000","last_change":"2024-04-15T15:15:10.295205+0000","last_active":"2024-04-15T15:15:15.865763+0000","last_peered":"2024-04-15T15:15:15.865763+0000","last_clean":"2024-04-15T15:15:15.865763+0000","last_became_active":"2024-04-15T15:15:10.294556+0000","last_became_peered":"2024-04-15T15:15:10.294556+0000","last_unstale":"2024-04-15T15:15:15.865763+0000","last_undegraded":"2024-04-15T15:15:15.865763+0000","last_fullsized":"2024-04-15T15:15:15.865763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864921+0000","last_change":"2024-04-15T15:15:10.297025+0000","last_active":"2024-04-15T15:15:15.864921+0000","last_peered":"2024-04-15T15:15:15.864921+0000","last_clean":"2024-04-15T15:15:15.864921+0000","last_became_active":"2024-04-15T15:15:10.296519+0000","last_became_peered":"2024-04-15T15:15:10.296519+0000","last_unstale":"2024-04-15T15:15:15.864921+0000","last_undegraded":"2024-04-15T15:15:15.864921+0000","last_fullsized":"2024-04-15T15:15:15.864921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393530+0000","last_change":"2024-04-15T15:15:10.288655+0000","last_active":"2024-04-15T15:15:15.393530+0000","last_peered":"2024-04-15T15:15:15.393530+0000","last_clean":"2024-04-15T15:15:15.393530+0000","last_became_active":"2024-04-15T15:15:10.286513+0000","last_became_peered":"2024-04-15T15:15:10.286513+0000","last_unstale":"2024-04-15T15:15:15.393530+0000","last_undegraded":"2024-04-15T15:15:15.393530+0000","last_fullsized":"2024-04-15T15:15:15.393530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865152+0000","last_change":"2024-04-15T15:15:10.299829+0000","last_active":"2024-04-15T15:15:15.865152+0000","last_peered":"2024-04-15T15:15:15.865152+0000","last_clean":"2024-04-15T15:15:15.865152+0000","last_became_active":"2024-04-15T15:15:10.299281+0000","last_became_peered":"2024-04-15T15:15:10.299281+0000","last_unstale":"2024-04-15T15:15:15.865152+0000","last_undegraded":"2024-04-15T15:15:15.865152+0000","last_fullsized":"2024-04-15T15:15:15.865152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:26:12.807802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308094+0000","last_change":"2024-04-15T15:15:10.301135+0000","last_active":"2024-04-15T15:15:15.308094+0000","last_peered":"2024-04-15T15:15:15.308094+0000","last_clean":"2024-04-15T15:15:15.308094+0000","last_became_active":"2024-04-15T15:15:10.300869+0000","last_became_peered":"2024-04-15T15:15:10.300869+0000","last_unstale":"2024-04-15T15:15:15.308094+0000","last_undegraded":"2024-04-15T15:15:15.308094+0000","last_fullsized":"2024-04-15T15:15:15.308094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309542+0000","last_change":"2024-04-15T15:15:10.285169+0000","last_active":"2024-04-15T15:15:15.309542+0000","last_peered":"2024-04-15T15:15:15.309542+0000","last_clean":"2024-04-15T15:15:15.309542+0000","last_became_active":"2024-04-15T15:15:10.284906+0000","last_became_peered":"2024-04-15T15:15:10.284906+0000","last_unstale":"2024-04-15T15:15:15.309542+0000","last_undegraded":"2024-04-15T15:15:15.309542+0000","last_fullsized":"2024-04-15T15:15:15.309542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:18.325913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310364+0000","last_change":"2024-04-15T15:15:10.300827+0000","last_active":"2024-04-15T15:15:15.310364+0000","last_peered":"2024-04-15T15:15:15.310364+0000","last_clean":"2024-04-15T15:15:15.310364+0000","last_became_active":"2024-04-15T15:15:10.300530+0000","last_became_peered":"2024-04-15T15:15:10.300530+0000","last_unstale":"2024-04-15T15:15:15.310364+0000","last_undegraded":"2024-04-15T15:15:15.310364+0000","last_fullsized":"2024-04-15T15:15:15.310364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310951+0000","last_change":"2024-04-15T15:15:10.297517+0000","last_active":"2024-04-15T15:15:15.310951+0000","last_peered":"2024-04-15T15:15:15.310951+0000","last_clean":"2024-04-15T15:15:15.310951+0000","last_became_active":"2024-04-15T15:15:10.297224+0000","last_became_peered":"2024-04-15T15:15:10.297224+0000","last_unstale":"2024-04-15T15:15:15.310951+0000","last_undegraded":"2024-04-15T15:15:15.310951+0000","last_fullsized":"2024-04-15T15:15:15.310951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866349+0000","last_change":"2024-04-15T15:15:10.305356+0000","last_active":"2024-04-15T15:15:15.866349+0000","last_peered":"2024-04-15T15:15:15.866349+0000","last_clean":"2024-04-15T15:15:15.866349+0000","last_became_active":"2024-04-15T15:15:10.305035+0000","last_became_peered":"2024-04-15T15:15:10.305035+0000","last_unstale":"2024-04-15T15:15:15.866349+0000","last_undegraded":"2024-04-15T15:15:15.866349+0000","last_fullsized":"2024-04-15T15:15:15.866349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:00:23.860901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308001+0000","last_change":"2024-04-15T15:15:10.299528+0000","last_active":"2024-04-15T15:15:15.308001+0000","last_peered":"2024-04-15T15:15:15.308001+0000","last_clean":"2024-04-15T15:15:15.308001+0000","last_became_active":"2024-04-15T15:15:10.299272+0000","last_became_peered":"2024-04-15T15:15:10.299272+0000","last_unstale":"2024-04-15T15:15:15.308001+0000","last_undegraded":"2024-04-15T15:15:15.308001+0000","last_fullsized":"2024-04-15T15:15:15.308001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865187+0000","last_change":"2024-04-15T15:15:10.295288+0000","last_active":"2024-04-15T15:15:15.865187+0000","last_peered":"2024-04-15T15:15:15.865187+0000","last_clean":"2024-04-15T15:15:15.865187+0000","last_became_active":"2024-04-15T15:15:10.295000+0000","last_became_peered":"2024-04-15T15:15:10.295000+0000","last_unstale":"2024-04-15T15:15:15.865187+0000","last_undegraded":"2024-04-15T15:15:15.865187+0000","last_fullsized":"2024-04-15T15:15:15.865187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864972+0000","last_change":"2024-04-15T15:15:10.298860+0000","last_active":"2024-04-15T15:15:15.864972+0000","last_peered":"2024-04-15T15:15:15.864972+0000","last_clean":"2024-04-15T15:15:15.864972+0000","last_became_active":"2024-04-15T15:15:10.296935+0000","last_became_peered":"2024-04-15T15:15:10.296935+0000","last_unstale":"2024-04-15T15:15:15.864972+0000","last_undegraded":"2024-04-15T15:15:15.864972+0000","last_fullsized":"2024-04-15T15:15:15.864972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865016+0000","last_change":"2024-04-15T15:15:10.296047+0000","last_active":"2024-04-15T15:15:15.865016+0000","last_peered":"2024-04-15T15:15:15.865016+0000","last_clean":"2024-04-15T15:15:15.865016+0000","last_became_active":"2024-04-15T15:15:10.295590+0000","last_became_peered":"2024-04-15T15:15:10.295590+0000","last_unstale":"2024-04-15T15:15:15.865016+0000","last_undegraded":"2024-04-15T15:15:15.865016+0000","last_fullsized":"2024-04-15T15:15:15.865016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865557+0000","last_change":"2024-04-15T15:15:10.298993+0000","last_active":"2024-04-15T15:15:15.865557+0000","last_peered":"2024-04-15T15:15:15.865557+0000","last_clean":"2024-04-15T15:15:15.865557+0000","last_became_active":"2024-04-15T15:15:10.298652+0000","last_became_peered":"2024-04-15T15:15:10.298652+0000","last_unstale":"2024-04-15T15:15:15.865557+0000","last_undegraded":"2024-04-15T15:15:15.865557+0000","last_fullsized":"2024-04-15T15:15:15.865557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865100+0000","last_change":"2024-04-15T15:15:10.291981+0000","last_active":"2024-04-15T15:15:15.865100+0000","last_peered":"2024-04-15T15:15:15.865100+0000","last_clean":"2024-04-15T15:15:15.865100+0000","last_became_active":"2024-04-15T15:15:10.291495+0000","last_became_peered":"2024-04-15T15:15:10.291495+0000","last_unstale":"2024-04-15T15:15:15.865100+0000","last_undegraded":"2024-04-15T15:15:15.865100+0000","last_fullsized":"2024-04-15T15:15:15.865100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394232+0000","last_change":"2024-04-15T15:15:10.288087+0000","last_active":"2024-04-15T15:15:15.394232+0000","last_peered":"2024-04-15T15:15:15.394232+0000","last_clean":"2024-04-15T15:15:15.394232+0000","last_became_active":"2024-04-15T15:15:10.287803+0000","last_became_peered":"2024-04-15T15:15:10.287803+0000","last_unstale":"2024-04-15T15:15:15.394232+0000","last_undegraded":"2024-04-15T15:15:15.394232+0000","last_fullsized":"2024-04-15T15:15:15.394232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866222+0000","last_change":"2024-04-15T15:15:10.302148+0000","last_active":"2024-04-15T15:15:15.866222+0000","last_peered":"2024-04-15T15:15:15.866222+0000","last_clean":"2024-04-15T15:15:15.866222+0000","last_became_active":"2024-04-15T15:15:10.301882+0000","last_became_peered":"2024-04-15T15:15:10.301882+0000","last_unstale":"2024-04-15T15:15:15.866222+0000","last_undegraded":"2024-04-15T15:15:15.866222+0000","last_fullsized":"2024-04-15T15:15:15.866222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":104773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":104807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":695653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:27.326 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:27.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.459+0000 7ff1dab67640 1 Processor -- start 2024-04-15T15:15:27.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1dab67640 1 -- start start 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1dab67640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff1d409b0e0 0x7ff1d40cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1dab67640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d40d0350 con 0x7ff1d415c8d0 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1dab67640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d40ce100 con 0x7ff1d409b650 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1dab67640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff1d40b5b00 con 0x7ff1d409b0e0 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1d8903640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff1d415c8d0 0x7ff1d409a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44688/0 (socket says 172.21.15.70:44688) 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.460+0000 7ff1d8903640 1 -- 172.21.15.70:0/2345804267 learned_addr learned my addr 172.21.15.70:0/2345804267 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d3fff640 1 --2- 172.21.15.70:0/2345804267 >> v2:172.21.15.96:3300/0 conn(0x7ff1d409b0e0 0x7ff1d40cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114464346 0 0) 0x7ff1d40d0350 con 0x7ff1d415c8d0 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1c0003520 con 0x7ff1d415c8d0 2024-04-15T15:15:27.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1138333011 0 0) 0x7ff1c0003520 con 0x7ff1d415c8d0 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 >> v2:172.21.15.96:3300/0 conn(0x7ff1d409b0e0 msgr2=0x7ff1d40cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 --2- 172.21.15.70:0/2345804267 >> v2:172.21.15.96:3300/0 conn(0x7ff1d409b0e0 0x7ff1d40cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 >> v1:172.21.15.96:6789/0 conn(0x7ff1d409b650 legacy=0x7ff1d40b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d40b54b0 con 0x7ff1d415c8d0 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7ff1c40019f0 con 0x7ff1d415c8d0 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 >> v1:172.21.15.70:6789/0 conn(0x7ff1d415c8d0 legacy=0x7ff1d409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1c0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1c0004800 0x7ff1c0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1c00051c0 0x7ff1c000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d40b5b00 con 0x7ff1c00039b0 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c000c060 con 0x7ff1c0004800 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c000c260 con 0x7ff1c00051c0 2024-04-15T15:15:27.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1c0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d3fff640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1c0004800 0x7ff1c0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d9104640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1c00051c0 0x7ff1c000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 -- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1c00051c0 msgr2=0x7ff1c000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1c00051c0 0x7ff1c000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 -- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1c0004800 msgr2=0x7ff1c0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1c0004800 0x7ff1c0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d8903640 1 -- 172.21.15.70:0/2345804267 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c0003520 con 0x7ff1c00039b0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.461+0000 7ff1d9104640 1 -- 172.21.15.70:0/2345804267 reap_dead start 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.462+0000 7ff1d8903640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1c0003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400a850 tx=0x7ff1c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994e286acf48d8ba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.462+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c400a4a0 con 0x7ff1c00039b0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.462+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c4034650 con 0x7ff1c00039b0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.462+0000 7ff1d37fe640 1 -- 172.21.15.70:0/2345804267 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4034a80 con 0x7ff1c00039b0 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 -- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 msgr2=0x7ff1c0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.467 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1c0003dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400a850 tx=0x7ff1c0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 -- 172.21.15.70:0/2345804267 shutdown_connections 2024-04-15T15:15:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 --2- 172.21.15.70:0/2345804267 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1c0003dd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 -- 172.21.15.70:0/2345804267 >> 172.21.15.70:0/2345804267 conn(0x7ff1d40b3fb0 msgr2=0x7ff1d40b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 -- 172.21.15.70:0/2345804267 shutdown_connections 2024-04-15T15:15:27.468 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.463+0000 7ff1dab67640 1 -- 172.21.15.70:0/2345804267 wait complete. 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.464+0000 7ff1dab67640 1 Processor -- start 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.464+0000 7ff1dab67640 1 -- start start 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1d4159610 0x7ff1d415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1d4156490 0x7ff1d41f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d40cdbf0 con 0x7ff1c00039b0 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff1d40cda70 con 0x7ff1d4159610 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1dab67640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff1d40cdd70 con 0x7ff1d4156490 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36490/0 (socket says 172.21.15.70:36490) 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 -- 172.21.15.70:0/895426656 learned_addr learned my addr 172.21.15.70:0/895426656 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d3fff640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1d4159610 0x7ff1d415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d9104640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1d4156490 0x7ff1d41f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 -- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1d4156490 msgr2=0x7ff1d41f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1d4156490 0x7ff1d41f4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 -- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1d4159610 msgr2=0x7ff1d415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1d4159610 0x7ff1d415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.465+0000 7ff1d8903640 1 -- 172.21.15.70:0/895426656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d4156930 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1d8903640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400a970 tx=0x7ff1c4001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4039040 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1c4043070 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d40cb200 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1c4004d60 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.466+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1d4156bd0 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.467+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1d4067af0 con 0x7ff1c00039b0 2024-04-15T15:15:27.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.467+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff1c4034650 con 0x7ff1c00039b0 2024-04-15T15:15:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.467+0000 7ff1d17fa640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.468+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff1c40addb0 con 0x7ff1c00039b0 2024-04-15T15:15:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.468+0000 7ff1d3fff640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.468+0000 7ff1d3fff640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 0x7ff1b406edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4156440 tx=0x7ff1bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.472+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff1d4067af0 con 0x7ff1c00039b0 2024-04-15T15:15:27.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.643+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff1d40690f0 con 0x7ff1b406c8f0 2024-04-15T15:15:27.653 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.651+0000 7ff1d17fa640 1 -- 172.21.15.70:0/895426656 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7ff1d40690f0 con 0x7ff1b406c8f0 2024-04-15T15:15:27.656 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:27.660 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:15:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.654+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 msgr2=0x7ff1b406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 0x7ff1b406edb0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4156440 tx=0x7ff1bc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 msgr2=0x7ff1d414a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff1c400a970 tx=0x7ff1c4001dc0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 shutdown_connections 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff1d4156490 0x7ff1d41f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff1d4159610 0x7ff1d415efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff1b406c8f0 0x7ff1b406edb0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 --2- 172.21.15.70:0/895426656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff1c00039b0 0x7ff1d414a0f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 >> 172.21.15.70:0/895426656 conn(0x7ff1d40b3fb0 msgr2=0x7ff1d415bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 shutdown_connections 2024-04-15T15:15:27.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.655+0000 7ff1dab67640 1 -- 172.21.15.70:0/895426656 wait complete. 2024-04-15T15:15:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-04-15T15:15:26.073286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594867,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236128,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738592,"statfs":{"total":773094113280,"available":772852318208,"internally_reserved":0,"allocated":3383296,"data_stored":2026910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2089,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.766019"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865551+0000","last_change":"2024-04-15T15:15:09.296489+0000","last_active":"2024-04-15T15:15:15.865551+0000","last_peered":"2024-04-15T15:15:15.865551+0000","last_clean":"2024-04-15T15:15:15.865551+0000","last_became_active":"2024-04-15T15:15:09.296225+0000","last_became_peered":"2024-04-15T15:15:09.296225+0000","last_unstale":"2024-04-15T15:15:15.865551+0000","last_undegraded":"2024-04-15T15:15:15.865551+0000","last_fullsized":"2024-04-15T15:15:15.865551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308730+0000","last_change":"2024-04-15T15:15:10.297175+0000","last_active":"2024-04-15T15:15:15.308730+0000","last_peered":"2024-04-15T15:15:15.308730+0000","last_clean":"2024-04-15T15:15:15.308730+0000","last_became_active":"2024-04-15T15:15:10.296707+0000","last_became_peered":"2024-04-15T15:15:10.296707+0000","last_unstale":"2024-04-15T15:15:15.308730+0000","last_undegraded":"2024-04-15T15:15:15.308730+0000","last_fullsized":"2024-04-15T15:15:15.308730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308030+0000","last_change":"2024-04-15T15:15:14.490094+0000","last_active":"2024-04-15T15:15:15.308030+0000","last_peered":"2024-04-15T15:15:15.308030+0000","last_clean":"2024-04-15T15:15:15.308030+0000","last_became_active":"2024-04-15T15:15:14.489727+0000","last_became_peered":"2024-04-15T15:15:14.489727+0000","last_unstale":"2024-04-15T15:15:15.308030+0000","last_undegraded":"2024-04-15T15:15:15.308030+0000","last_fullsized":"2024-04-15T15:15:15.308030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308822+0000","last_change":"2024-04-15T15:15:09.279298+0000","last_active":"2024-04-15T15:15:15.308822+0000","last_peered":"2024-04-15T15:15:15.308822+0000","last_clean":"2024-04-15T15:15:15.308822+0000","last_became_active":"2024-04-15T15:15:09.279018+0000","last_became_peered":"2024-04-15T15:15:09.279018+0000","last_unstale":"2024-04-15T15:15:15.308822+0000","last_undegraded":"2024-04-15T15:15:15.308822+0000","last_fullsized":"2024-04-15T15:15:15.308822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865283+0000","last_change":"2024-04-15T15:15:10.300836+0000","last_active":"2024-04-15T15:15:15.865283+0000","last_peered":"2024-04-15T15:15:15.865283+0000","last_clean":"2024-04-15T15:15:15.865283+0000","last_became_active":"2024-04-15T15:15:10.300574+0000","last_became_peered":"2024-04-15T15:15:10.300574+0000","last_unstale":"2024-04-15T15:15:15.865283+0000","last_undegraded":"2024-04-15T15:15:15.865283+0000","last_fullsized":"2024-04-15T15:15:15.865283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310208+0000","last_change":"2024-04-15T15:15:14.327665+0000","last_active":"2024-04-15T15:15:15.310208+0000","last_peered":"2024-04-15T15:15:15.310208+0000","last_clean":"2024-04-15T15:15:15.310208+0000","last_became_active":"2024-04-15T15:15:14.327161+0000","last_became_peered":"2024-04-15T15:15:14.327161+0000","last_unstale":"2024-04-15T15:15:15.310208+0000","last_undegraded":"2024-04-15T15:15:15.310208+0000","last_fullsized":"2024-04-15T15:15:15.310208+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866087+0000","last_change":"2024-04-15T15:15:09.297431+0000","last_active":"2024-04-15T15:15:15.866087+0000","last_peered":"2024-04-15T15:15:15.866087+0000","last_clean":"2024-04-15T15:15:15.866087+0000","last_became_active":"2024-04-15T15:15:09.297182+0000","last_became_peered":"2024-04-15T15:15:09.297182+0000","last_unstale":"2024-04-15T15:15:15.866087+0000","last_undegraded":"2024-04-15T15:15:15.866087+0000","last_fullsized":"2024-04-15T15:15:15.866087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:45.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865971+0000","last_change":"2024-04-15T15:15:10.303063+0000","last_active":"2024-04-15T15:15:15.865971+0000","last_peered":"2024-04-15T15:15:15.865971+0000","last_clean":"2024-04-15T15:15:15.865971+0000","last_became_active":"2024-04-15T15:15:10.302785+0000","last_became_peered":"2024-04-15T15:15:10.302785+0000","last_unstale":"2024-04-15T15:15:15.865971+0000","last_undegraded":"2024-04-15T15:15:15.865971+0000","last_fullsized":"2024-04-15T15:15:15.865971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309364+0000","last_change":"2024-04-15T15:15:14.328804+0000","last_active":"2024-04-15T15:15:15.309364+0000","last_peered":"2024-04-15T15:15:15.309364+0000","last_clean":"2024-04-15T15:15:15.309364+0000","last_became_active":"2024-04-15T15:15:14.327886+0000","last_became_peered":"2024-04-15T15:15:14.327886+0000","last_unstale":"2024-04-15T15:15:15.309364+0000","last_undegraded":"2024-04-15T15:15:15.309364+0000","last_fullsized":"2024-04-15T15:15:15.309364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:25.064499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865605+0000","last_change":"2024-04-15T15:15:09.300461+0000","last_active":"2024-04-15T15:15:15.865605+0000","last_peered":"2024-04-15T15:15:15.865605+0000","last_clean":"2024-04-15T15:15:15.865605+0000","last_became_active":"2024-04-15T15:15:09.300159+0000","last_became_peered":"2024-04-15T15:15:09.300159+0000","last_unstale":"2024-04-15T15:15:15.865605+0000","last_undegraded":"2024-04-15T15:15:15.865605+0000","last_fullsized":"2024-04-15T15:15:15.865605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865784+0000","last_change":"2024-04-15T15:15:10.298282+0000","last_active":"2024-04-15T15:15:15.865784+0000","last_peered":"2024-04-15T15:15:15.865784+0000","last_clean":"2024-04-15T15:15:15.865784+0000","last_became_active":"2024-04-15T15:15:10.297734+0000","last_became_peered":"2024-04-15T15:15:10.297734+0000","last_unstale":"2024-04-15T15:15:15.865784+0000","last_undegraded":"2024-04-15T15:15:15.865784+0000","last_fullsized":"2024-04-15T15:15:15.865784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309722+0000","last_change":"2024-04-15T15:15:14.321155+0000","last_active":"2024-04-15T15:15:15.309722+0000","last_peered":"2024-04-15T15:15:15.309722+0000","last_clean":"2024-04-15T15:15:15.309722+0000","last_became_active":"2024-04-15T15:15:14.320777+0000","last_became_peered":"2024-04-15T15:15:14.320777+0000","last_unstale":"2024-04-15T15:15:15.309722+0000","last_undegraded":"2024-04-15T15:15:15.309722+0000","last_fullsized":"2024-04-15T15:15:15.309722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:36.915564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310304+0000","last_change":"2024-04-15T15:15:10.288957+0000","last_active":"2024-04-15T15:15:15.310304+0000","last_peered":"2024-04-15T15:15:15.310304+0000","last_clean":"2024-04-15T15:15:15.310304+0000","last_became_active":"2024-04-15T15:15:10.288692+0000","last_became_peered":"2024-04-15T15:15:10.288692+0000","last_unstale":"2024-04-15T15:15:15.310304+0000","last_undegraded":"2024-04-15T15:15:15.310304+0000","last_fullsized":"2024-04-15T15:15:15.310304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865797+0000","last_change":"2024-04-15T15:15:09.295590+0000","last_active":"2024-04-15T15:15:15.865797+0000","last_peered":"2024-04-15T15:15:15.865797+0000","last_clean":"2024-04-15T15:15:15.865797+0000","last_became_active":"2024-04-15T15:15:09.295295+0000","last_became_peered":"2024-04-15T15:15:09.295295+0000","last_unstale":"2024-04-15T15:15:15.865797+0000","last_undegraded":"2024-04-15T15:15:15.865797+0000","last_fullsized":"2024-04-15T15:15:15.865797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309855+0000","last_change":"2024-04-15T15:15:14.327835+0000","last_active":"2024-04-15T15:15:15.309855+0000","last_peered":"2024-04-15T15:15:15.309855+0000","last_clean":"2024-04-15T15:15:15.309855+0000","last_became_active":"2024-04-15T15:15:14.327245+0000","last_became_peered":"2024-04-15T15:15:14.327245+0000","last_unstale":"2024-04-15T15:15:15.309855+0000","last_undegraded":"2024-04-15T15:15:15.309855+0000","last_fullsized":"2024-04-15T15:15:15.309855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308188+0000","last_change":"2024-04-15T15:15:10.295527+0000","last_active":"2024-04-15T15:15:15.308188+0000","last_peered":"2024-04-15T15:15:15.308188+0000","last_clean":"2024-04-15T15:15:15.308188+0000","last_became_active":"2024-04-15T15:15:10.295265+0000","last_became_peered":"2024-04-15T15:15:10.295265+0000","last_unstale":"2024-04-15T15:15:15.308188+0000","last_undegraded":"2024-04-15T15:15:15.308188+0000","last_fullsized":"2024-04-15T15:15:15.308188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393749+0000","last_change":"2024-04-15T15:15:09.291151+0000","last_active":"2024-04-15T15:15:15.393749+0000","last_peered":"2024-04-15T15:15:15.393749+0000","last_clean":"2024-04-15T15:15:15.393749+0000","last_became_active":"2024-04-15T15:15:09.290851+0000","last_became_peered":"2024-04-15T15:15:09.290851+0000","last_unstale":"2024-04-15T15:15:15.393749+0000","last_undegraded":"2024-04-15T15:15:15.393749+0000","last_fullsized":"2024-04-15T15:15:15.393749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309375+0000","last_change":"2024-04-15T15:15:14.327866+0000","last_active":"2024-04-15T15:15:15.309375+0000","last_peered":"2024-04-15T15:15:15.309375+0000","last_clean":"2024-04-15T15:15:15.309375+0000","last_became_active":"2024-04-15T15:15:14.327398+0000","last_became_peered":"2024-04-15T15:15:14.327398+0000","last_unstale":"2024-04-15T15:15:15.309375+0000","last_undegraded":"2024-04-15T15:15:15.309375+0000","last_fullsized":"2024-04-15T15:15:15.309375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:45.919286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308185+0000","last_change":"2024-04-15T15:15:10.294406+0000","last_active":"2024-04-15T15:15:15.308185+0000","last_peered":"2024-04-15T15:15:15.308185+0000","last_clean":"2024-04-15T15:15:15.308185+0000","last_became_active":"2024-04-15T15:15:10.294005+0000","last_became_peered":"2024-04-15T15:15:10.294005+0000","last_unstale":"2024-04-15T15:15:15.308185+0000","last_undegraded":"2024-04-15T15:15:15.308185+0000","last_fullsized":"2024-04-15T15:15:15.308185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393556+0000","last_change":"2024-04-15T15:15:09.288397+0000","last_active":"2024-04-15T15:15:15.393556+0000","last_peered":"2024-04-15T15:15:15.393556+0000","last_clean":"2024-04-15T15:15:15.393556+0000","last_became_active":"2024-04-15T15:15:09.288135+0000","last_became_peered":"2024-04-15T15:15:09.288135+0000","last_unstale":"2024-04-15T15:15:15.393556+0000","last_undegraded":"2024-04-15T15:15:15.393556+0000","last_fullsized":"2024-04-15T15:15:15.393556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394110+0000","last_change":"2024-04-15T15:15:14.488972+0000","last_active":"2024-04-15T15:15:15.394110+0000","last_peered":"2024-04-15T15:15:15.394110+0000","last_clean":"2024-04-15T15:15:15.394110+0000","last_became_active":"2024-04-15T15:15:14.488302+0000","last_became_peered":"2024-04-15T15:15:14.488302+0000","last_unstale":"2024-04-15T15:15:15.394110+0000","last_undegraded":"2024-04-15T15:15:15.394110+0000","last_fullsized":"2024-04-15T15:15:15.394110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310517+0000","last_change":"2024-04-15T15:15:10.294428+0000","last_active":"2024-04-15T15:15:15.310517+0000","last_peered":"2024-04-15T15:15:15.310517+0000","last_clean":"2024-04-15T15:15:15.310517+0000","last_became_active":"2024-04-15T15:15:10.294121+0000","last_became_peered":"2024-04-15T15:15:10.294121+0000","last_unstale":"2024-04-15T15:15:15.310517+0000","last_undegraded":"2024-04-15T15:15:15.310517+0000","last_fullsized":"2024-04-15T15:15:15.310517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:06.364679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865099+0000","last_change":"2024-04-15T15:15:09.294900+0000","last_active":"2024-04-15T15:15:15.865099+0000","last_peered":"2024-04-15T15:15:15.865099+0000","last_clean":"2024-04-15T15:15:15.865099+0000","last_became_active":"2024-04-15T15:15:09.294601+0000","last_became_peered":"2024-04-15T15:15:09.294601+0000","last_unstale":"2024-04-15T15:15:15.865099+0000","last_undegraded":"2024-04-15T15:15:15.865099+0000","last_fullsized":"2024-04-15T15:15:15.865099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309437+0000","last_change":"2024-04-15T15:15:14.328664+0000","last_active":"2024-04-15T15:15:15.309437+0000","last_peered":"2024-04-15T15:15:15.309437+0000","last_clean":"2024-04-15T15:15:15.309437+0000","last_became_active":"2024-04-15T15:15:14.327796+0000","last_became_peered":"2024-04-15T15:15:14.327796+0000","last_unstale":"2024-04-15T15:15:15.309437+0000","last_undegraded":"2024-04-15T15:15:15.309437+0000","last_fullsized":"2024-04-15T15:15:15.309437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:28.892025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865447+0000","last_change":"2024-04-15T15:15:10.288747+0000","last_active":"2024-04-15T15:15:15.865447+0000","last_peered":"2024-04-15T15:15:15.865447+0000","last_clean":"2024-04-15T15:15:15.865447+0000","last_became_active":"2024-04-15T15:15:10.288465+0000","last_became_peered":"2024-04-15T15:15:10.288465+0000","last_unstale":"2024-04-15T15:15:15.865447+0000","last_undegraded":"2024-04-15T15:15:15.865447+0000","last_fullsized":"2024-04-15T15:15:15.865447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311197+0000","last_change":"2024-04-15T15:15:09.291999+0000","last_active":"2024-04-15T15:15:15.311197+0000","last_peered":"2024-04-15T15:15:15.311197+0000","last_clean":"2024-04-15T15:15:15.311197+0000","last_became_active":"2024-04-15T15:15:09.291744+0000","last_became_peered":"2024-04-15T15:15:09.291744+0000","last_unstale":"2024-04-15T15:15:15.311197+0000","last_undegraded":"2024-04-15T15:15:15.311197+0000","last_fullsized":"2024-04-15T15:15:15.311197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310618+0000","last_change":"2024-04-15T15:15:14.491129+0000","last_active":"2024-04-15T15:15:15.310618+0000","last_peered":"2024-04-15T15:15:15.310618+0000","last_clean":"2024-04-15T15:15:15.310618+0000","last_became_active":"2024-04-15T15:15:14.490619+0000","last_became_peered":"2024-04-15T15:15:14.490619+0000","last_unstale":"2024-04-15T15:15:15.310618+0000","last_undegraded":"2024-04-15T15:15:15.310618+0000","last_fullsized":"2024-04-15T15:15:15.310618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:46.083837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308841+0000","last_change":"2024-04-15T15:15:10.297516+0000","last_active":"2024-04-15T15:15:15.308841+0000","last_peered":"2024-04-15T15:15:15.308841+0000","last_clean":"2024-04-15T15:15:15.308841+0000","last_became_active":"2024-04-15T15:15:10.297282+0000","last_became_peered":"2024-04-15T15:15:10.297282+0000","last_unstale":"2024-04-15T15:15:15.308841+0000","last_undegraded":"2024-04-15T15:15:15.308841+0000","last_fullsized":"2024-04-15T15:15:15.308841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394314+0000","last_change":"2024-04-15T15:15:09.288389+0000","last_active":"2024-04-15T15:15:15.394314+0000","last_peered":"2024-04-15T15:15:15.394314+0000","last_clean":"2024-04-15T15:15:15.394314+0000","last_became_active":"2024-04-15T15:15:09.288101+0000","last_became_peered":"2024-04-15T15:15:09.288101+0000","last_unstale":"2024-04-15T15:15:15.394314+0000","last_undegraded":"2024-04-15T15:15:15.394314+0000","last_fullsized":"2024-04-15T15:15:15.394314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.307926+0000","last_change":"2024-04-15T15:15:14.488661+0000","last_active":"2024-04-15T15:15:15.307926+0000","last_peered":"2024-04-15T15:15:15.307926+0000","last_clean":"2024-04-15T15:15:15.307926+0000","last_became_active":"2024-04-15T15:15:14.487944+0000","last_became_peered":"2024-04-15T15:15:14.487944+0000","last_unstale":"2024-04-15T15:15:15.307926+0000","last_undegraded":"2024-04-15T15:15:15.307926+0000","last_fullsized":"2024-04-15T15:15:15.307926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:36.475872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393929+0000","last_change":"2024-04-15T15:15:10.289924+0000","last_active":"2024-04-15T15:15:15.393929+0000","last_peered":"2024-04-15T15:15:15.393929+0000","last_clean":"2024-04-15T15:15:15.393929+0000","last_became_active":"2024-04-15T15:15:10.289662+0000","last_became_peered":"2024-04-15T15:15:10.289662+0000","last_unstale":"2024-04-15T15:15:15.393929+0000","last_undegraded":"2024-04-15T15:15:15.393929+0000","last_fullsized":"2024-04-15T15:15:15.393929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866240+0000","last_change":"2024-04-15T15:15:09.295866+0000","last_active":"2024-04-15T15:15:15.866240+0000","last_peered":"2024-04-15T15:15:15.866240+0000","last_clean":"2024-04-15T15:15:15.866240+0000","last_became_active":"2024-04-15T15:15:09.295507+0000","last_became_peered":"2024-04-15T15:15:09.295507+0000","last_unstale":"2024-04-15T15:15:15.866240+0000","last_undegraded":"2024-04-15T15:15:15.866240+0000","last_fullsized":"2024-04-15T15:15:15.866240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:01.918524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865520+0000","last_change":"2024-04-15T15:15:14.487233+0000","last_active":"2024-04-15T15:15:15.865520+0000","last_peered":"2024-04-15T15:15:15.865520+0000","last_clean":"2024-04-15T15:15:15.865520+0000","last_became_active":"2024-04-15T15:15:14.486846+0000","last_became_peered":"2024-04-15T15:15:14.486846+0000","last_unstale":"2024-04-15T15:15:15.865520+0000","last_undegraded":"2024-04-15T15:15:15.865520+0000","last_fullsized":"2024-04-15T15:15:15.865520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865899+0000","last_change":"2024-04-15T15:15:10.299154+0000","last_active":"2024-04-15T15:15:15.865899+0000","last_peered":"2024-04-15T15:15:15.865899+0000","last_clean":"2024-04-15T15:15:15.865899+0000","last_became_active":"2024-04-15T15:15:10.298711+0000","last_became_peered":"2024-04-15T15:15:10.298711+0000","last_unstale":"2024-04-15T15:15:15.865899+0000","last_undegraded":"2024-04-15T15:15:15.865899+0000","last_fullsized":"2024-04-15T15:15:15.865899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394042+0000","last_change":"2024-04-15T15:15:09.289347+0000","last_active":"2024-04-15T15:15:15.394042+0000","last_peered":"2024-04-15T15:15:15.394042+0000","last_clean":"2024-04-15T15:15:15.394042+0000","last_became_active":"2024-04-15T15:15:09.289062+0000","last_became_peered":"2024-04-15T15:15:09.289062+0000","last_unstale":"2024-04-15T15:15:15.394042+0000","last_undegraded":"2024-04-15T15:15:15.394042+0000","last_fullsized":"2024-04-15T15:15:15.394042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393588+0000","last_change":"2024-04-15T15:15:14.330539+0000","last_active":"2024-04-15T15:15:15.393588+0000","last_peered":"2024-04-15T15:15:15.393588+0000","last_clean":"2024-04-15T15:15:15.393588+0000","last_became_active":"2024-04-15T15:15:14.330173+0000","last_became_peered":"2024-04-15T15:15:14.330173+0000","last_unstale":"2024-04-15T15:15:15.393588+0000","last_undegraded":"2024-04-15T15:15:15.393588+0000","last_fullsized":"2024-04-15T15:15:15.393588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310032+0000","last_change":"2024-04-15T15:15:10.286603+0000","last_active":"2024-04-15T15:15:15.310032+0000","last_peered":"2024-04-15T15:15:15.310032+0000","last_clean":"2024-04-15T15:15:15.310032+0000","last_became_active":"2024-04-15T15:15:10.286306+0000","last_became_peered":"2024-04-15T15:15:10.286306+0000","last_unstale":"2024-04-15T15:15:15.310032+0000","last_undegraded":"2024-04-15T15:15:15.310032+0000","last_fullsized":"2024-04-15T15:15:15.310032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:47.785526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865601+0000","last_change":"2024-04-15T15:15:09.291918+0000","last_active":"2024-04-15T15:15:15.865601+0000","last_peered":"2024-04-15T15:15:15.865601+0000","last_clean":"2024-04-15T15:15:15.865601+0000","last_became_active":"2024-04-15T15:15:09.291664+0000","last_became_peered":"2024-04-15T15:15:09.291664+0000","last_unstale":"2024-04-15T15:15:15.865601+0000","last_undegraded":"2024-04-15T15:15:15.865601+0000","last_fullsized":"2024-04-15T15:15:15.865601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393983+0000","last_change":"2024-04-15T15:15:14.329421+0000","last_active":"2024-04-15T15:15:15.393983+0000","last_peered":"2024-04-15T15:15:15.393983+0000","last_clean":"2024-04-15T15:15:15.393983+0000","last_became_active":"2024-04-15T15:15:14.329120+0000","last_became_peered":"2024-04-15T15:15:14.329120+0000","last_unstale":"2024-04-15T15:15:15.393983+0000","last_undegraded":"2024-04-15T15:15:15.393983+0000","last_fullsized":"2024-04-15T15:15:15.393983+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393608+0000","last_change":"2024-04-15T15:15:10.290334+0000","last_active":"2024-04-15T15:15:15.393608+0000","last_peered":"2024-04-15T15:15:15.393608+0000","last_clean":"2024-04-15T15:15:15.393608+0000","last_became_active":"2024-04-15T15:15:10.290075+0000","last_became_peered":"2024-04-15T15:15:10.290075+0000","last_unstale":"2024-04-15T15:15:15.393608+0000","last_undegraded":"2024-04-15T15:15:15.393608+0000","last_fullsized":"2024-04-15T15:15:15.393608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864908+0000","last_change":"2024-04-15T15:15:09.291564+0000","last_active":"2024-04-15T15:15:15.864908+0000","last_peered":"2024-04-15T15:15:15.864908+0000","last_clean":"2024-04-15T15:15:15.864908+0000","last_became_active":"2024-04-15T15:15:09.291265+0000","last_became_peered":"2024-04-15T15:15:09.291265+0000","last_unstale":"2024-04-15T15:15:15.864908+0000","last_undegraded":"2024-04-15T15:15:15.864908+0000","last_fullsized":"2024-04-15T15:15:15.864908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394137+0000","last_change":"2024-04-15T15:15:14.329535+0000","last_active":"2024-04-15T15:15:15.394137+0000","last_peered":"2024-04-15T15:15:15.394137+0000","last_clean":"2024-04-15T15:15:15.394137+0000","last_became_active":"2024-04-15T15:15:14.329173+0000","last_became_peered":"2024-04-15T15:15:14.329173+0000","last_unstale":"2024-04-15T15:15:15.394137+0000","last_undegraded":"2024-04-15T15:15:15.394137+0000","last_fullsized":"2024-04-15T15:15:15.394137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864715+0000","last_change":"2024-04-15T15:15:10.292048+0000","last_active":"2024-04-15T15:15:15.864715+0000","last_peered":"2024-04-15T15:15:15.864715+0000","last_clean":"2024-04-15T15:15:15.864715+0000","last_became_active":"2024-04-15T15:15:10.291773+0000","last_became_peered":"2024-04-15T15:15:10.291773+0000","last_unstale":"2024-04-15T15:15:15.864715+0000","last_undegraded":"2024-04-15T15:15:15.864715+0000","last_fullsized":"2024-04-15T15:15:15.864715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:45.215354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865214+0000","last_change":"2024-04-15T15:15:09.295426+0000","last_active":"2024-04-15T15:15:15.865214+0000","last_peered":"2024-04-15T15:15:15.865214+0000","last_clean":"2024-04-15T15:15:15.865214+0000","last_became_active":"2024-04-15T15:15:09.295096+0000","last_became_peered":"2024-04-15T15:15:09.295096+0000","last_unstale":"2024-04-15T15:15:15.865214+0000","last_undegraded":"2024-04-15T15:15:15.865214+0000","last_fullsized":"2024-04-15T15:15:15.865214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865346+0000","last_change":"2024-04-15T15:15:14.487282+0000","last_active":"2024-04-15T15:15:15.865346+0000","last_peered":"2024-04-15T15:15:15.865346+0000","last_clean":"2024-04-15T15:15:15.865346+0000","last_became_active":"2024-04-15T15:15:14.486873+0000","last_became_peered":"2024-04-15T15:15:14.486873+0000","last_unstale":"2024-04-15T15:15:15.865346+0000","last_undegraded":"2024-04-15T15:15:15.865346+0000","last_fullsized":"2024-04-15T15:15:15.865346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308149+0000","last_change":"2024-04-15T15:15:10.301516+0000","last_active":"2024-04-15T15:15:15.308149+0000","last_peered":"2024-04-15T15:15:15.308149+0000","last_clean":"2024-04-15T15:15:15.308149+0000","last_became_active":"2024-04-15T15:15:10.301272+0000","last_became_peered":"2024-04-15T15:15:10.301272+0000","last_unstale":"2024-04-15T15:15:15.308149+0000","last_undegraded":"2024-04-15T15:15:15.308149+0000","last_fullsized":"2024-04-15T15:15:15.308149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865920+0000","last_change":"2024-04-15T15:15:09.292652+0000","last_active":"2024-04-15T15:15:15.865920+0000","last_peered":"2024-04-15T15:15:15.865920+0000","last_clean":"2024-04-15T15:15:15.865920+0000","last_became_active":"2024-04-15T15:15:09.292366+0000","last_became_peered":"2024-04-15T15:15:09.292366+0000","last_unstale":"2024-04-15T15:15:15.865920+0000","last_undegraded":"2024-04-15T15:15:15.865920+0000","last_fullsized":"2024-04-15T15:15:15.865920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865328+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:15:15.865328+0000","last_peered":"2024-04-15T15:15:15.865328+0000","last_clean":"2024-04-15T15:15:15.865328+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:15:15.865328+0000","last_undegraded":"2024-04-15T15:15:15.865328+0000","last_fullsized":"2024-04-15T15:15:15.865328+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310218+0000","last_change":"2024-04-15T15:15:14.488973+0000","last_active":"2024-04-15T15:15:15.310218+0000","last_peered":"2024-04-15T15:15:15.310218+0000","last_clean":"2024-04-15T15:15:15.310218+0000","last_became_active":"2024-04-15T15:15:14.488151+0000","last_became_peered":"2024-04-15T15:15:14.488151+0000","last_unstale":"2024-04-15T15:15:15.310218+0000","last_undegraded":"2024-04-15T15:15:15.310218+0000","last_fullsized":"2024-04-15T15:15:15.310218+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865992+0000","last_change":"2024-04-15T15:15:10.303487+0000","last_active":"2024-04-15T15:15:15.865992+0000","last_peered":"2024-04-15T15:15:15.865992+0000","last_clean":"2024-04-15T15:15:15.865992+0000","last_became_active":"2024-04-15T15:15:10.303222+0000","last_became_peered":"2024-04-15T15:15:10.303222+0000","last_unstale":"2024-04-15T15:15:15.865992+0000","last_undegraded":"2024-04-15T15:15:15.865992+0000","last_fullsized":"2024-04-15T15:15:15.865992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:09.292763+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:09.292482+0000","last_became_peered":"2024-04-15T15:15:09.292482+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310239+0000","last_change":"2024-04-15T15:15:14.487139+0000","last_active":"2024-04-15T15:15:15.310239+0000","last_peered":"2024-04-15T15:15:15.310239+0000","last_clean":"2024-04-15T15:15:15.310239+0000","last_became_active":"2024-04-15T15:15:14.486746+0000","last_became_peered":"2024-04-15T15:15:14.486746+0000","last_unstale":"2024-04-15T15:15:15.310239+0000","last_undegraded":"2024-04-15T15:15:15.310239+0000","last_fullsized":"2024-04-15T15:15:15.310239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865734+0000","last_change":"2024-04-15T15:15:10.298528+0000","last_active":"2024-04-15T15:15:15.865734+0000","last_peered":"2024-04-15T15:15:15.865734+0000","last_clean":"2024-04-15T15:15:15.865734+0000","last_became_active":"2024-04-15T15:15:10.298175+0000","last_became_peered":"2024-04-15T15:15:10.298175+0000","last_unstale":"2024-04-15T15:15:15.865734+0000","last_undegraded":"2024-04-15T15:15:15.865734+0000","last_fullsized":"2024-04-15T15:15:15.865734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394078+0000","last_change":"2024-04-15T15:15:09.287274+0000","last_active":"2024-04-15T15:15:15.394078+0000","last_peered":"2024-04-15T15:15:15.394078+0000","last_clean":"2024-04-15T15:15:15.394078+0000","last_became_active":"2024-04-15T15:15:09.287022+0000","last_became_peered":"2024-04-15T15:15:09.287022+0000","last_unstale":"2024-04-15T15:15:15.394078+0000","last_undegraded":"2024-04-15T15:15:15.394078+0000","last_fullsized":"2024-04-15T15:15:15.394078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865617+0000","last_change":"2024-04-15T15:15:14.327367+0000","last_active":"2024-04-15T15:15:15.865617+0000","last_peered":"2024-04-15T15:15:15.865617+0000","last_clean":"2024-04-15T15:15:15.865617+0000","last_became_active":"2024-04-15T15:15:14.327012+0000","last_became_peered":"2024-04-15T15:15:14.327012+0000","last_unstale":"2024-04-15T15:15:15.865617+0000","last_undegraded":"2024-04-15T15:15:15.865617+0000","last_fullsized":"2024-04-15T15:15:15.865617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866362+0000","last_change":"2024-04-15T15:15:10.300664+0000","last_active":"2024-04-15T15:15:15.866362+0000","last_peered":"2024-04-15T15:15:15.866362+0000","last_clean":"2024-04-15T15:15:15.866362+0000","last_became_active":"2024-04-15T15:15:10.300411+0000","last_became_peered":"2024-04-15T15:15:10.300411+0000","last_unstale":"2024-04-15T15:15:15.866362+0000","last_undegraded":"2024-04-15T15:15:15.866362+0000","last_fullsized":"2024-04-15T15:15:15.866362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:32.450402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864760+0000","last_change":"2024-04-15T15:15:09.291588+0000","last_active":"2024-04-15T15:15:15.864760+0000","last_peered":"2024-04-15T15:15:15.864760+0000","last_clean":"2024-04-15T15:15:15.864760+0000","last_became_active":"2024-04-15T15:15:09.291317+0000","last_became_peered":"2024-04-15T15:15:09.291317+0000","last_unstale":"2024-04-15T15:15:15.864760+0000","last_undegraded":"2024-04-15T15:15:15.864760+0000","last_fullsized":"2024-04-15T15:15:15.864760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:14.488073+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:14.487630+0000","last_became_peered":"2024-04-15T15:15:14.487630+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866072+0000","last_change":"2024-04-15T15:15:10.293789+0000","last_active":"2024-04-15T15:15:15.866072+0000","last_peered":"2024-04-15T15:15:15.866072+0000","last_clean":"2024-04-15T15:15:15.866072+0000","last_became_active":"2024-04-15T15:15:10.293508+0000","last_became_peered":"2024-04-15T15:15:10.293508+0000","last_unstale":"2024-04-15T15:15:15.866072+0000","last_undegraded":"2024-04-15T15:15:15.866072+0000","last_fullsized":"2024-04-15T15:15:15.866072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:13.731730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310273+0000","last_change":"2024-04-15T15:15:09.280627+0000","last_active":"2024-04-15T15:15:15.310273+0000","last_peered":"2024-04-15T15:15:15.310273+0000","last_clean":"2024-04-15T15:15:15.310273+0000","last_became_active":"2024-04-15T15:15:09.280327+0000","last_became_peered":"2024-04-15T15:15:09.280327+0000","last_unstale":"2024-04-15T15:15:15.310273+0000","last_undegraded":"2024-04-15T15:15:15.310273+0000","last_fullsized":"2024-04-15T15:15:15.310273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:24.741048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308985+0000","last_change":"2024-04-15T15:15:14.327503+0000","last_active":"2024-04-15T15:15:15.308985+0000","last_peered":"2024-04-15T15:15:15.308985+0000","last_clean":"2024-04-15T15:15:15.308985+0000","last_became_active":"2024-04-15T15:15:14.327045+0000","last_became_peered":"2024-04-15T15:15:14.327045+0000","last_unstale":"2024-04-15T15:15:15.308985+0000","last_undegraded":"2024-04-15T15:15:15.308985+0000","last_fullsized":"2024-04-15T15:15:15.308985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:41.707546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865253+0000","last_change":"2024-04-15T15:15:10.294511+0000","last_active":"2024-04-15T15:15:15.865253+0000","last_peered":"2024-04-15T15:15:15.865253+0000","last_clean":"2024-04-15T15:15:15.865253+0000","last_became_active":"2024-04-15T15:15:10.294217+0000","last_became_peered":"2024-04-15T15:15:10.294217+0000","last_unstale":"2024-04-15T15:15:15.865253+0000","last_undegraded":"2024-04-15T15:15:15.865253+0000","last_fullsized":"2024-04-15T15:15:15.865253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864350+0000","last_change":"2024-04-15T15:15:09.289902+0000","last_active":"2024-04-15T15:15:15.864350+0000","last_peered":"2024-04-15T15:15:15.864350+0000","last_clean":"2024-04-15T15:15:15.864350+0000","last_became_active":"2024-04-15T15:15:09.289521+0000","last_became_peered":"2024-04-15T15:15:09.289521+0000","last_unstale":"2024-04-15T15:15:15.864350+0000","last_undegraded":"2024-04-15T15:15:15.864350+0000","last_fullsized":"2024-04-15T15:15:15.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865201+0000","last_change":"2024-04-15T15:15:14.325169+0000","last_active":"2024-04-15T15:15:15.865201+0000","last_peered":"2024-04-15T15:15:15.865201+0000","last_clean":"2024-04-15T15:15:15.865201+0000","last_became_active":"2024-04-15T15:15:14.324680+0000","last_became_peered":"2024-04-15T15:15:14.324680+0000","last_unstale":"2024-04-15T15:15:15.865201+0000","last_undegraded":"2024-04-15T15:15:15.865201+0000","last_fullsized":"2024-04-15T15:15:15.865201+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308937+0000","last_change":"2024-04-15T15:15:10.291452+0000","last_active":"2024-04-15T15:15:15.308937+0000","last_peered":"2024-04-15T15:15:15.308937+0000","last_clean":"2024-04-15T15:15:15.308937+0000","last_became_active":"2024-04-15T15:15:10.291170+0000","last_became_peered":"2024-04-15T15:15:10.291170+0000","last_unstale":"2024-04-15T15:15:15.308937+0000","last_undegraded":"2024-04-15T15:15:15.308937+0000","last_fullsized":"2024-04-15T15:15:15.308937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309951+0000","last_change":"2024-04-15T15:15:09.292208+0000","last_active":"2024-04-15T15:15:15.309951+0000","last_peered":"2024-04-15T15:15:15.309951+0000","last_clean":"2024-04-15T15:15:15.309951+0000","last_became_active":"2024-04-15T15:15:09.290702+0000","last_became_peered":"2024-04-15T15:15:09.290702+0000","last_unstale":"2024-04-15T15:15:15.309951+0000","last_undegraded":"2024-04-15T15:15:15.309951+0000","last_fullsized":"2024-04-15T15:15:15.309951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308793+0000","last_change":"2024-04-15T15:15:14.488657+0000","last_active":"2024-04-15T15:15:15.308793+0000","last_peered":"2024-04-15T15:15:15.308793+0000","last_clean":"2024-04-15T15:15:15.308793+0000","last_became_active":"2024-04-15T15:15:14.488282+0000","last_became_peered":"2024-04-15T15:15:14.488282+0000","last_unstale":"2024-04-15T15:15:15.308793+0000","last_undegraded":"2024-04-15T15:15:15.308793+0000","last_fullsized":"2024-04-15T15:15:15.308793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:21:03.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865696+0000","last_change":"2024-04-15T15:15:10.301823+0000","last_active":"2024-04-15T15:15:15.865696+0000","last_peered":"2024-04-15T15:15:15.865696+0000","last_clean":"2024-04-15T15:15:15.865696+0000","last_became_active":"2024-04-15T15:15:10.301545+0000","last_became_peered":"2024-04-15T15:15:10.301545+0000","last_unstale":"2024-04-15T15:15:15.865696+0000","last_undegraded":"2024-04-15T15:15:15.865696+0000","last_fullsized":"2024-04-15T15:15:15.865696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865481+0000","last_change":"2024-04-15T15:15:09.291724+0000","last_active":"2024-04-15T15:15:15.865481+0000","last_peered":"2024-04-15T15:15:15.865481+0000","last_clean":"2024-04-15T15:15:15.865481+0000","last_became_active":"2024-04-15T15:15:09.291390+0000","last_became_peered":"2024-04-15T15:15:09.291390+0000","last_unstale":"2024-04-15T15:15:15.865481+0000","last_undegraded":"2024-04-15T15:15:15.865481+0000","last_fullsized":"2024-04-15T15:15:15.865481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308524+0000","last_change":"2024-04-15T15:15:14.491287+0000","last_active":"2024-04-15T15:15:15.308524+0000","last_peered":"2024-04-15T15:15:15.308524+0000","last_clean":"2024-04-15T15:15:15.308524+0000","last_became_active":"2024-04-15T15:15:14.488643+0000","last_became_peered":"2024-04-15T15:15:14.488643+0000","last_unstale":"2024-04-15T15:15:15.308524+0000","last_undegraded":"2024-04-15T15:15:15.308524+0000","last_fullsized":"2024-04-15T15:15:15.308524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:29:47.252030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865184+0000","last_change":"2024-04-15T15:15:10.293687+0000","last_active":"2024-04-15T15:15:15.865184+0000","last_peered":"2024-04-15T15:15:15.865184+0000","last_clean":"2024-04-15T15:15:15.865184+0000","last_became_active":"2024-04-15T15:15:10.293430+0000","last_became_peered":"2024-04-15T15:15:10.293430+0000","last_unstale":"2024-04-15T15:15:15.865184+0000","last_undegraded":"2024-04-15T15:15:15.865184+0000","last_fullsized":"2024-04-15T15:15:15.865184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865653+0000","last_change":"2024-04-15T15:15:09.282027+0000","last_active":"2024-04-15T15:15:15.865653+0000","last_peered":"2024-04-15T15:15:15.865653+0000","last_clean":"2024-04-15T15:15:15.865653+0000","last_became_active":"2024-04-15T15:15:09.281730+0000","last_became_peered":"2024-04-15T15:15:09.281730+0000","last_unstale":"2024-04-15T15:15:15.865653+0000","last_undegraded":"2024-04-15T15:15:15.865653+0000","last_fullsized":"2024-04-15T15:15:15.865653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:01.954710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866059+0000","last_change":"2024-04-15T15:15:14.489189+0000","last_active":"2024-04-15T15:15:15.866059+0000","last_peered":"2024-04-15T15:15:15.866059+0000","last_clean":"2024-04-15T15:15:15.866059+0000","last_became_active":"2024-04-15T15:15:14.488758+0000","last_became_peered":"2024-04-15T15:15:14.488758+0000","last_unstale":"2024-04-15T15:15:15.866059+0000","last_undegraded":"2024-04-15T15:15:15.866059+0000","last_fullsized":"2024-04-15T15:15:15.866059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865898+0000","last_change":"2024-04-15T15:15:10.298047+0000","last_active":"2024-04-15T15:15:15.865898+0000","last_peered":"2024-04-15T15:15:15.865898+0000","last_clean":"2024-04-15T15:15:15.865898+0000","last_became_active":"2024-04-15T15:15:10.297779+0000","last_became_peered":"2024-04-15T15:15:10.297779+0000","last_unstale":"2024-04-15T15:15:15.865898+0000","last_undegraded":"2024-04-15T15:15:15.865898+0000","last_fullsized":"2024-04-15T15:15:15.865898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:19.312104+0000","last_change":"2024-04-15T15:15:09.290413+0000","last_active":"2024-04-15T15:15:19.312104+0000","last_peered":"2024-04-15T15:15:19.312104+0000","last_clean":"2024-04-15T15:15:19.312104+0000","last_became_active":"2024-04-15T15:15:09.289971+0000","last_became_peered":"2024-04-15T15:15:09.289971+0000","last_unstale":"2024-04-15T15:15:19.312104+0000","last_undegraded":"2024-04-15T15:15:19.312104+0000","last_fullsized":"2024-04-15T15:15:19.312104+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3701,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393701+0000","last_change":"2024-04-15T15:15:14.324351+0000","last_active":"2024-04-15T15:15:15.393701+0000","last_peered":"2024-04-15T15:15:15.393701+0000","last_clean":"2024-04-15T15:15:15.393701+0000","last_became_active":"2024-04-15T15:15:14.322177+0000","last_became_peered":"2024-04-15T15:15:14.322177+0000","last_unstale":"2024-04-15T15:15:15.393701+0000","last_undegraded":"2024-04-15T15:15:15.393701+0000","last_fullsized":"2024-04-15T15:15:15.393701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310092+0000","last_change":"2024-04-15T15:15:10.290888+0000","last_active":"2024-04-15T15:15:15.310092+0000","last_peered":"2024-04-15T15:15:15.310092+0000","last_clean":"2024-04-15T15:15:15.310092+0000","last_became_active":"2024-04-15T15:15:10.290630+0000","last_became_peered":"2024-04-15T15:15:10.290630+0000","last_unstale":"2024-04-15T15:15:15.310092+0000","last_undegraded":"2024-04-15T15:15:15.310092+0000","last_fullsized":"2024-04-15T15:15:15.310092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394037+0000","last_change":"2024-04-15T15:15:09.291950+0000","last_active":"2024-04-15T15:15:15.394037+0000","last_peered":"2024-04-15T15:15:15.394037+0000","last_clean":"2024-04-15T15:15:15.394037+0000","last_became_active":"2024-04-15T15:15:09.291687+0000","last_became_peered":"2024-04-15T15:15:09.291687+0000","last_unstale":"2024-04-15T15:15:15.394037+0000","last_undegraded":"2024-04-15T15:15:15.394037+0000","last_fullsized":"2024-04-15T15:15:15.394037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309744+0000","last_change":"2024-04-15T15:15:14.328229+0000","last_active":"2024-04-15T15:15:15.309744+0000","last_peered":"2024-04-15T15:15:15.309744+0000","last_clean":"2024-04-15T15:15:15.309744+0000","last_became_active":"2024-04-15T15:15:14.327167+0000","last_became_peered":"2024-04-15T15:15:14.327167+0000","last_unstale":"2024-04-15T15:15:15.309744+0000","last_undegraded":"2024-04-15T15:15:15.309744+0000","last_fullsized":"2024-04-15T15:15:15.309744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:36.054042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865433+0000","last_change":"2024-04-15T15:15:10.300821+0000","last_active":"2024-04-15T15:15:15.865433+0000","last_peered":"2024-04-15T15:15:15.865433+0000","last_clean":"2024-04-15T15:15:15.865433+0000","last_became_active":"2024-04-15T15:15:10.300561+0000","last_became_peered":"2024-04-15T15:15:10.300561+0000","last_unstale":"2024-04-15T15:15:15.865433+0000","last_undegraded":"2024-04-15T15:15:15.865433+0000","last_fullsized":"2024-04-15T15:15:15.865433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865315+0000","last_change":"2024-04-15T15:15:09.294709+0000","last_active":"2024-04-15T15:15:15.865315+0000","last_peered":"2024-04-15T15:15:15.865315+0000","last_clean":"2024-04-15T15:15:15.865315+0000","last_became_active":"2024-04-15T15:15:09.294135+0000","last_became_peered":"2024-04-15T15:15:09.294135+0000","last_unstale":"2024-04-15T15:15:15.865315+0000","last_undegraded":"2024-04-15T15:15:15.865315+0000","last_fullsized":"2024-04-15T15:15:15.865315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308127+0000","last_change":"2024-04-15T15:15:14.332302+0000","last_active":"2024-04-15T15:15:15.308127+0000","last_peered":"2024-04-15T15:15:15.308127+0000","last_clean":"2024-04-15T15:15:15.308127+0000","last_became_active":"2024-04-15T15:15:14.331893+0000","last_became_peered":"2024-04-15T15:15:14.331893+0000","last_unstale":"2024-04-15T15:15:15.308127+0000","last_undegraded":"2024-04-15T15:15:15.308127+0000","last_fullsized":"2024-04-15T15:15:15.308127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:18:57.970154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865270+0000","last_change":"2024-04-15T15:15:10.295887+0000","last_active":"2024-04-15T15:15:15.865270+0000","last_peered":"2024-04-15T15:15:15.865270+0000","last_clean":"2024-04-15T15:15:15.865270+0000","last_became_active":"2024-04-15T15:15:10.295591+0000","last_became_peered":"2024-04-15T15:15:10.295591+0000","last_unstale":"2024-04-15T15:15:15.865270+0000","last_undegraded":"2024-04-15T15:15:15.865270+0000","last_fullsized":"2024-04-15T15:15:15.865270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311117+0000","last_change":"2024-04-15T15:15:09.290489+0000","last_active":"2024-04-15T15:15:15.311117+0000","last_peered":"2024-04-15T15:15:15.311117+0000","last_clean":"2024-04-15T15:15:15.311117+0000","last_became_active":"2024-04-15T15:15:09.290207+0000","last_became_peered":"2024-04-15T15:15:09.290207+0000","last_unstale":"2024-04-15T15:15:15.311117+0000","last_undegraded":"2024-04-15T15:15:15.311117+0000","last_fullsized":"2024-04-15T15:15:15.311117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308129+0000","last_change":"2024-04-15T15:15:14.490596+0000","last_active":"2024-04-15T15:15:15.308129+0000","last_peered":"2024-04-15T15:15:15.308129+0000","last_clean":"2024-04-15T15:15:15.308129+0000","last_became_active":"2024-04-15T15:15:14.489609+0000","last_became_peered":"2024-04-15T15:15:14.489609+0000","last_unstale":"2024-04-15T15:15:15.308129+0000","last_undegraded":"2024-04-15T15:15:15.308129+0000","last_fullsized":"2024-04-15T15:15:15.308129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865063+0000","last_change":"2024-04-15T15:15:14.489631+0000","last_active":"2024-04-15T15:15:15.865063+0000","last_peered":"2024-04-15T15:15:15.865063+0000","last_clean":"2024-04-15T15:15:15.865063+0000","last_became_active":"2024-04-15T15:15:14.489248+0000","last_became_peered":"2024-04-15T15:15:14.489248+0000","last_unstale":"2024-04-15T15:15:15.865063+0000","last_undegraded":"2024-04-15T15:15:15.865063+0000","last_fullsized":"2024-04-15T15:15:15.865063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393445+0000","last_change":"2024-04-15T15:15:10.293423+0000","last_active":"2024-04-15T15:15:15.393445+0000","last_peered":"2024-04-15T15:15:15.393445+0000","last_clean":"2024-04-15T15:15:15.393445+0000","last_became_active":"2024-04-15T15:15:10.293101+0000","last_became_peered":"2024-04-15T15:15:10.293101+0000","last_unstale":"2024-04-15T15:15:15.393445+0000","last_undegraded":"2024-04-15T15:15:15.393445+0000","last_fullsized":"2024-04-15T15:15:15.393445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865159+0000","last_change":"2024-04-15T15:15:09.290248+0000","last_active":"2024-04-15T15:15:15.865159+0000","last_peered":"2024-04-15T15:15:15.865159+0000","last_clean":"2024-04-15T15:15:15.865159+0000","last_became_active":"2024-04-15T15:15:09.289792+0000","last_became_peered":"2024-04-15T15:15:09.289792+0000","last_unstale":"2024-04-15T15:15:15.865159+0000","last_undegraded":"2024-04-15T15:15:15.865159+0000","last_fullsized":"2024-04-15T15:15:15.865159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311080+0000","last_change":"2024-04-15T15:15:14.323489+0000","last_active":"2024-04-15T15:15:15.311080+0000","last_peered":"2024-04-15T15:15:15.311080+0000","last_clean":"2024-04-15T15:15:15.311080+0000","last_became_active":"2024-04-15T15:15:14.322956+0000","last_became_peered":"2024-04-15T15:15:14.322956+0000","last_unstale":"2024-04-15T15:15:15.311080+0000","last_undegraded":"2024-04-15T15:15:15.311080+0000","last_fullsized":"2024-04-15T15:15:15.311080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308115+0000","last_change":"2024-04-15T15:15:10.295446+0000","last_active":"2024-04-15T15:15:15.308115+0000","last_peered":"2024-04-15T15:15:15.308115+0000","last_clean":"2024-04-15T15:15:15.308115+0000","last_became_active":"2024-04-15T15:15:10.295165+0000","last_became_peered":"2024-04-15T15:15:10.295165+0000","last_unstale":"2024-04-15T15:15:15.308115+0000","last_undegraded":"2024-04-15T15:15:15.308115+0000","last_fullsized":"2024-04-15T15:15:15.308115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308300+0000","last_change":"2024-04-15T15:15:09.289277+0000","last_active":"2024-04-15T15:15:15.308300+0000","last_peered":"2024-04-15T15:15:15.308300+0000","last_clean":"2024-04-15T15:15:15.308300+0000","last_became_active":"2024-04-15T15:15:09.289026+0000","last_became_peered":"2024-04-15T15:15:09.289026+0000","last_unstale":"2024-04-15T15:15:15.308300+0000","last_undegraded":"2024-04-15T15:15:15.308300+0000","last_fullsized":"2024-04-15T15:15:15.308300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310503+0000","last_change":"2024-04-15T15:15:09.285504+0000","last_active":"2024-04-15T15:15:15.310503+0000","last_peered":"2024-04-15T15:15:15.310503+0000","last_clean":"2024-04-15T15:15:15.310503+0000","last_became_active":"2024-04-15T15:15:09.285219+0000","last_became_peered":"2024-04-15T15:15:09.285219+0000","last_unstale":"2024-04-15T15:15:15.310503+0000","last_undegraded":"2024-04-15T15:15:15.310503+0000","last_fullsized":"2024-04-15T15:15:15.310503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865651+0000","last_change":"2024-04-15T15:15:10.297720+0000","last_active":"2024-04-15T15:15:15.865651+0000","last_peered":"2024-04-15T15:15:15.865651+0000","last_clean":"2024-04-15T15:15:15.865651+0000","last_became_active":"2024-04-15T15:15:10.297469+0000","last_became_peered":"2024-04-15T15:15:10.297469+0000","last_unstale":"2024-04-15T15:15:15.865651+0000","last_undegraded":"2024-04-15T15:15:15.865651+0000","last_fullsized":"2024-04-15T15:15:15.865651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864649+0000","last_change":"2024-04-15T15:15:14.327342+0000","last_active":"2024-04-15T15:15:15.864649+0000","last_peered":"2024-04-15T15:15:15.864649+0000","last_clean":"2024-04-15T15:15:15.864649+0000","last_became_active":"2024-04-15T15:15:14.326929+0000","last_became_peered":"2024-04-15T15:15:14.326929+0000","last_unstale":"2024-04-15T15:15:15.864649+0000","last_undegraded":"2024-04-15T15:15:15.864649+0000","last_fullsized":"2024-04-15T15:15:15.864649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:19:56.492223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308298+0000","last_change":"2024-04-15T15:15:09.288401+0000","last_active":"2024-04-15T15:15:15.308298+0000","last_peered":"2024-04-15T15:15:15.308298+0000","last_clean":"2024-04-15T15:15:15.308298+0000","last_became_active":"2024-04-15T15:15:09.288124+0000","last_became_peered":"2024-04-15T15:15:09.288124+0000","last_unstale":"2024-04-15T15:15:15.308298+0000","last_undegraded":"2024-04-15T15:15:15.308298+0000","last_fullsized":"2024-04-15T15:15:15.308298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309493+0000","last_change":"2024-04-15T15:15:10.293787+0000","last_active":"2024-04-15T15:15:15.309493+0000","last_peered":"2024-04-15T15:15:15.309493+0000","last_clean":"2024-04-15T15:15:15.309493+0000","last_became_active":"2024-04-15T15:15:10.293409+0000","last_became_peered":"2024-04-15T15:15:10.293409+0000","last_unstale":"2024-04-15T15:15:15.309493+0000","last_undegraded":"2024-04-15T15:15:15.309493+0000","last_fullsized":"2024-04-15T15:15:15.309493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:30.764771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308821+0000","last_change":"2024-04-15T15:15:14.323853+0000","last_active":"2024-04-15T15:15:15.308821+0000","last_peered":"2024-04-15T15:15:15.308821+0000","last_clean":"2024-04-15T15:15:15.308821+0000","last_became_active":"2024-04-15T15:15:14.323547+0000","last_became_peered":"2024-04-15T15:15:14.323547+0000","last_unstale":"2024-04-15T15:15:15.308821+0000","last_undegraded":"2024-04-15T15:15:15.308821+0000","last_fullsized":"2024-04-15T15:15:15.308821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:48.237751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308815+0000","last_change":"2024-04-15T15:15:10.290641+0000","last_active":"2024-04-15T15:15:15.308815+0000","last_peered":"2024-04-15T15:15:15.308815+0000","last_clean":"2024-04-15T15:15:15.308815+0000","last_became_active":"2024-04-15T15:15:10.290342+0000","last_became_peered":"2024-04-15T15:15:10.290342+0000","last_unstale":"2024-04-15T15:15:15.308815+0000","last_undegraded":"2024-04-15T15:15:15.308815+0000","last_fullsized":"2024-04-15T15:15:15.308815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864807+0000","last_change":"2024-04-15T15:15:10.302121+0000","last_active":"2024-04-15T15:15:15.864807+0000","last_peered":"2024-04-15T15:15:15.864807+0000","last_clean":"2024-04-15T15:15:15.864807+0000","last_became_active":"2024-04-15T15:15:10.301860+0000","last_became_peered":"2024-04-15T15:15:10.301860+0000","last_unstale":"2024-04-15T15:15:15.864807+0000","last_undegraded":"2024-04-15T15:15:15.864807+0000","last_fullsized":"2024-04-15T15:15:15.864807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865207+0000","last_change":"2024-04-15T15:15:10.298052+0000","last_active":"2024-04-15T15:15:15.865207+0000","last_peered":"2024-04-15T15:15:15.865207+0000","last_clean":"2024-04-15T15:15:15.865207+0000","last_became_active":"2024-04-15T15:15:10.296656+0000","last_became_peered":"2024-04-15T15:15:10.296656+0000","last_unstale":"2024-04-15T15:15:15.865207+0000","last_undegraded":"2024-04-15T15:15:15.865207+0000","last_fullsized":"2024-04-15T15:15:15.865207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865604+0000","last_change":"2024-04-15T15:15:10.296027+0000","last_active":"2024-04-15T15:15:15.865604+0000","last_peered":"2024-04-15T15:15:15.865604+0000","last_clean":"2024-04-15T15:15:15.865604+0000","last_became_active":"2024-04-15T15:15:10.295741+0000","last_became_peered":"2024-04-15T15:15:10.295741+0000","last_unstale":"2024-04-15T15:15:15.865604+0000","last_undegraded":"2024-04-15T15:15:15.865604+0000","last_fullsized":"2024-04-15T15:15:15.865604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308554+0000","last_change":"2024-04-15T15:15:10.287343+0000","last_active":"2024-04-15T15:15:15.308554+0000","last_peered":"2024-04-15T15:15:15.308554+0000","last_clean":"2024-04-15T15:15:15.308554+0000","last_became_active":"2024-04-15T15:15:10.287082+0000","last_became_peered":"2024-04-15T15:15:10.287082+0000","last_unstale":"2024-04-15T15:15:15.308554+0000","last_undegraded":"2024-04-15T15:15:15.308554+0000","last_fullsized":"2024-04-15T15:15:15.308554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393974+0000","last_change":"2024-04-15T15:15:10.289020+0000","last_active":"2024-04-15T15:15:15.393974+0000","last_peered":"2024-04-15T15:15:15.393974+0000","last_clean":"2024-04-15T15:15:15.393974+0000","last_became_active":"2024-04-15T15:15:10.288777+0000","last_became_peered":"2024-04-15T15:15:10.288777+0000","last_unstale":"2024-04-15T15:15:15.393974+0000","last_undegraded":"2024-04-15T15:15:15.393974+0000","last_fullsized":"2024-04-15T15:15:15.393974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310906+0000","last_change":"2024-04-15T15:15:10.290594+0000","last_active":"2024-04-15T15:15:15.310906+0000","last_peered":"2024-04-15T15:15:15.310906+0000","last_clean":"2024-04-15T15:15:15.310906+0000","last_became_active":"2024-04-15T15:15:10.290345+0000","last_became_peered":"2024-04-15T15:15:10.290345+0000","last_unstale":"2024-04-15T15:15:15.310906+0000","last_undegraded":"2024-04-15T15:15:15.310906+0000","last_fullsized":"2024-04-15T15:15:15.310906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308747+0000","last_change":"2024-04-15T15:15:10.297500+0000","last_active":"2024-04-15T15:15:15.308747+0000","last_peered":"2024-04-15T15:15:15.308747+0000","last_clean":"2024-04-15T15:15:15.308747+0000","last_became_active":"2024-04-15T15:15:10.296730+0000","last_became_peered":"2024-04-15T15:15:10.296730+0000","last_unstale":"2024-04-15T15:15:15.308747+0000","last_undegraded":"2024-04-15T15:15:15.308747+0000","last_fullsized":"2024-04-15T15:15:15.308747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865052+0000","last_change":"2024-04-15T15:15:10.296225+0000","last_active":"2024-04-15T15:15:15.865052+0000","last_peered":"2024-04-15T15:15:15.865052+0000","last_clean":"2024-04-15T15:15:15.865052+0000","last_became_active":"2024-04-15T15:15:10.295930+0000","last_became_peered":"2024-04-15T15:15:10.295930+0000","last_unstale":"2024-04-15T15:15:15.865052+0000","last_undegraded":"2024-04-15T15:15:15.865052+0000","last_fullsized":"2024-04-15T15:15:15.865052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308196+0000","last_change":"2024-04-15T15:15:10.289427+0000","last_active":"2024-04-15T15:15:15.308196+0000","last_peered":"2024-04-15T15:15:15.308196+0000","last_clean":"2024-04-15T15:15:15.308196+0000","last_became_active":"2024-04-15T15:15:10.289170+0000","last_became_peered":"2024-04-15T15:15:10.289170+0000","last_unstale":"2024-04-15T15:15:15.308196+0000","last_undegraded":"2024-04-15T15:15:15.308196+0000","last_fullsized":"2024-04-15T15:15:15.308196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865103+0000","last_change":"2024-04-15T15:15:10.304028+0000","last_active":"2024-04-15T15:15:15.865103+0000","last_peered":"2024-04-15T15:15:15.865103+0000","last_clean":"2024-04-15T15:15:15.865103+0000","last_became_active":"2024-04-15T15:15:10.303771+0000","last_became_peered":"2024-04-15T15:15:10.303771+0000","last_unstale":"2024-04-15T15:15:15.865103+0000","last_undegraded":"2024-04-15T15:15:15.865103+0000","last_fullsized":"2024-04-15T15:15:15.865103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865533+0000","last_change":"2024-04-15T15:15:10.293639+0000","last_active":"2024-04-15T15:15:15.865533+0000","last_peered":"2024-04-15T15:15:15.865533+0000","last_clean":"2024-04-15T15:15:15.865533+0000","last_became_active":"2024-04-15T15:15:10.293220+0000","last_became_peered":"2024-04-15T15:15:10.293220+0000","last_unstale":"2024-04-15T15:15:15.865533+0000","last_undegraded":"2024-04-15T15:15:15.865533+0000","last_fullsized":"2024-04-15T15:15:15.865533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310554+0000","last_change":"2024-04-15T15:15:10.289725+0000","last_active":"2024-04-15T15:15:15.310554+0000","last_peered":"2024-04-15T15:15:15.310554+0000","last_clean":"2024-04-15T15:15:15.310554+0000","last_became_active":"2024-04-15T15:15:10.289378+0000","last_became_peered":"2024-04-15T15:15:10.289378+0000","last_unstale":"2024-04-15T15:15:15.310554+0000","last_undegraded":"2024-04-15T15:15:15.310554+0000","last_fullsized":"2024-04-15T15:15:15.310554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866173+0000","last_change":"2024-04-15T15:15:10.295231+0000","last_active":"2024-04-15T15:15:15.866173+0000","last_peered":"2024-04-15T15:15:15.866173+0000","last_clean":"2024-04-15T15:15:15.866173+0000","last_became_active":"2024-04-15T15:15:10.294974+0000","last_became_peered":"2024-04-15T15:15:10.294974+0000","last_unstale":"2024-04-15T15:15:15.866173+0000","last_undegraded":"2024-04-15T15:15:15.866173+0000","last_fullsized":"2024-04-15T15:15:15.866173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864735+0000","last_change":"2024-04-15T15:15:10.300143+0000","last_active":"2024-04-15T15:15:15.864735+0000","last_peered":"2024-04-15T15:15:15.864735+0000","last_clean":"2024-04-15T15:15:15.864735+0000","last_became_active":"2024-04-15T15:15:10.299889+0000","last_became_peered":"2024-04-15T15:15:10.299889+0000","last_unstale":"2024-04-15T15:15:15.864735+0000","last_undegraded":"2024-04-15T15:15:15.864735+0000","last_fullsized":"2024-04-15T15:15:15.864735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:46.735175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865763+0000","last_change":"2024-04-15T15:15:10.295205+0000","last_active":"2024-04-15T15:15:15.865763+0000","last_peered":"2024-04-15T15:15:15.865763+0000","last_clean":"2024-04-15T15:15:15.865763+0000","last_became_active":"2024-04-15T15:15:10.294556+0000","last_became_peered":"2024-04-15T15:15:10.294556+0000","last_unstale":"2024-04-15T15:15:15.865763+0000","last_undegraded":"2024-04-15T15:15:15.865763+0000","last_fullsized":"2024-04-15T15:15:15.865763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864921+0000","last_change":"2024-04-15T15:15:10.297025+0000","last_active":"2024-04-15T15:15:15.864921+0000","last_peered":"2024-04-15T15:15:15.864921+0000","last_clean":"2024-04-15T15:15:15.864921+0000","last_became_active":"2024-04-15T15:15:10.296519+0000","last_became_peered":"2024-04-15T15:15:10.296519+0000","last_unstale":"2024-04-15T15:15:15.864921+0000","last_undegraded":"2024-04-15T15:15:15.864921+0000","last_fullsized":"2024-04-15T15:15:15.864921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393530+0000","last_change":"2024-04-15T15:15:10.288655+0000","last_active":"2024-04-15T15:15:15.393530+0000","last_peered":"2024-04-15T15:15:15.393530+0000","last_clean":"2024-04-15T15:15:15.393530+0000","last_became_active":"2024-04-15T15:15:10.286513+0000","last_became_peered":"2024-04-15T15:15:10.286513+0000","last_unstale":"2024-04-15T15:15:15.393530+0000","last_undegraded":"2024-04-15T15:15:15.393530+0000","last_fullsized":"2024-04-15T15:15:15.393530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865152+0000","last_change":"2024-04-15T15:15:10.299829+0000","last_active":"2024-04-15T15:15:15.865152+0000","last_peered":"2024-04-15T15:15:15.865152+0000","last_clean":"2024-04-15T15:15:15.865152+0000","last_became_active":"2024-04-15T15:15:10.299281+0000","last_became_peered":"2024-04-15T15:15:10.299281+0000","last_unstale":"2024-04-15T15:15:15.865152+0000","last_undegraded":"2024-04-15T15:15:15.865152+0000","last_fullsized":"2024-04-15T15:15:15.865152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:26:12.807802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308094+0000","last_change":"2024-04-15T15:15:10.301135+0000","last_active":"2024-04-15T15:15:15.308094+0000","last_peered":"2024-04-15T15:15:15.308094+0000","last_clean":"2024-04-15T15:15:15.308094+0000","last_became_active":"2024-04-15T15:15:10.300869+0000","last_became_peered":"2024-04-15T15:15:10.300869+0000","last_unstale":"2024-04-15T15:15:15.308094+0000","last_undegraded":"2024-04-15T15:15:15.308094+0000","last_fullsized":"2024-04-15T15:15:15.308094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309542+0000","last_change":"2024-04-15T15:15:10.285169+0000","last_active":"2024-04-15T15:15:15.309542+0000","last_peered":"2024-04-15T15:15:15.309542+0000","last_clean":"2024-04-15T15:15:15.309542+0000","last_became_active":"2024-04-15T15:15:10.284906+0000","last_became_peered":"2024-04-15T15:15:10.284906+0000","last_unstale":"2024-04-15T15:15:15.309542+0000","last_undegraded":"2024-04-15T15:15:15.309542+0000","last_fullsized":"2024-04-15T15:15:15.309542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:18.325913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310364+0000","last_change":"2024-04-15T15:15:10.300827+0000","last_active":"2024-04-15T15:15:15.310364+0000","last_peered":"2024-04-15T15:15:15.310364+0000","last_clean":"2024-04-15T15:15:15.310364+0000","last_became_active":"2024-04-15T15:15:10.300530+0000","last_became_peered":"2024-04-15T15:15:10.300530+0000","last_unstale":"2024-04-15T15:15:15.310364+0000","last_undegraded":"2024-04-15T15:15:15.310364+0000","last_fullsized":"2024-04-15T15:15:15.310364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310951+0000","last_change":"2024-04-15T15:15:10.297517+0000","last_active":"2024-04-15T15:15:15.310951+0000","last_peered":"2024-04-15T15:15:15.310951+0000","last_clean":"2024-04-15T15:15:15.310951+0000","last_became_active":"2024-04-15T15:15:10.297224+0000","last_became_peered":"2024-04-15T15:15:10.297224+0000","last_unstale":"2024-04-15T15:15:15.310951+0000","last_undegraded":"2024-04-15T15:15:15.310951+0000","last_fullsized":"2024-04-15T15:15:15.310951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866349+0000","last_change":"2024-04-15T15:15:10.305356+0000","last_active":"2024-04-15T15:15:15.866349+0000","last_peered":"2024-04-15T15:15:15.866349+0000","last_clean":"2024-04-15T15:15:15.866349+0000","last_became_active":"2024-04-15T15:15:10.305035+0000","last_became_peered":"2024-04-15T15:15:10.305035+0000","last_unstale":"2024-04-15T15:15:15.866349+0000","last_undegraded":"2024-04-15T15:15:15.866349+0000","last_fullsized":"2024-04-15T15:15:15.866349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:00:23.860901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308001+0000","last_change":"2024-04-15T15:15:10.299528+0000","last_active":"2024-04-15T15:15:15.308001+0000","last_peered":"2024-04-15T15:15:15.308001+0000","last_clean":"2024-04-15T15:15:15.308001+0000","last_became_active":"2024-04-15T15:15:10.299272+0000","last_became_peered":"2024-04-15T15:15:10.299272+0000","last_unstale":"2024-04-15T15:15:15.308001+0000","last_undegraded":"2024-04-15T15:15:15.308001+0000","last_fullsized":"2024-04-15T15:15:15.308001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865187+0000","last_change":"2024-04-15T15:15:10.295288+0000","last_active":"2024-04-15T15:15:15.865187+0000","last_peered":"2024-04-15T15:15:15.865187+0000","last_clean":"2024-04-15T15:15:15.865187+0000","last_became_active":"2024-04-15T15:15:10.295000+0000","last_became_peered":"2024-04-15T15:15:10.295000+0000","last_unstale":"2024-04-15T15:15:15.865187+0000","last_undegraded":"2024-04-15T15:15:15.865187+0000","last_fullsized":"2024-04-15T15:15:15.865187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864972+0000","last_change":"2024-04-15T15:15:10.298860+0000","last_active":"2024-04-15T15:15:15.864972+0000","last_peered":"2024-04-15T15:15:15.864972+0000","last_clean":"2024-04-15T15:15:15.864972+0000","last_became_active":"2024-04-15T15:15:10.296935+0000","last_became_peered":"2024-04-15T15:15:10.296935+0000","last_unstale":"2024-04-15T15:15:15.864972+0000","last_undegraded":"2024-04-15T15:15:15.864972+0000","last_fullsized":"2024-04-15T15:15:15.864972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865016+0000","last_change":"2024-04-15T15:15:10.296047+0000","last_active":"2024-04-15T15:15:15.865016+0000","last_peered":"2024-04-15T15:15:15.865016+0000","last_clean":"2024-04-15T15:15:15.865016+0000","last_became_active":"2024-04-15T15:15:10.295590+0000","last_became_peered":"2024-04-15T15:15:10.295590+0000","last_unstale":"2024-04-15T15:15:15.865016+0000","last_undegraded":"2024-04-15T15:15:15.865016+0000","last_fullsized":"2024-04-15T15:15:15.865016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865557+0000","last_change":"2024-04-15T15:15:10.298993+0000","last_active":"2024-04-15T15:15:15.865557+0000","last_peered":"2024-04-15T15:15:15.865557+0000","last_clean":"2024-04-15T15:15:15.865557+0000","last_became_active":"2024-04-15T15:15:10.298652+0000","last_became_peered":"2024-04-15T15:15:10.298652+0000","last_unstale":"2024-04-15T15:15:15.865557+0000","last_undegraded":"2024-04-15T15:15:15.865557+0000","last_fullsized":"2024-04-15T15:15:15.865557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865100+0000","last_change":"2024-04-15T15:15:10.291981+0000","last_active":"2024-04-15T15:15:15.865100+0000","last_peered":"2024-04-15T15:15:15.865100+0000","last_clean":"2024-04-15T15:15:15.865100+0000","last_became_active":"2024-04-15T15:15:10.291495+0000","last_became_peered":"2024-04-15T15:15:10.291495+0000","last_unstale":"2024-04-15T15:15:15.865100+0000","last_undegraded":"2024-04-15T15:15:15.865100+0000","last_fullsized":"2024-04-15T15:15:15.865100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394232+0000","last_change":"2024-04-15T15:15:10.288087+0000","last_active":"2024-04-15T15:15:15.394232+0000","last_peered":"2024-04-15T15:15:15.394232+0000","last_clean":"2024-04-15T15:15:15.394232+0000","last_became_active":"2024-04-15T15:15:10.287803+0000","last_became_peered":"2024-04-15T15:15:10.287803+0000","last_unstale":"2024-04-15T15:15:15.394232+0000","last_undegraded":"2024-04-15T15:15:15.394232+0000","last_fullsized":"2024-04-15T15:15:15.394232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866222+0000","last_change":"2024-04-15T15:15:10.302148+0000","last_active":"2024-04-15T15:15:15.866222+0000","last_peered":"2024-04-15T15:15:15.866222+0000","last_clean":"2024-04-15T15:15:15.866222+0000","last_became_active":"2024-04-15T15:15:10.301882+0000","last_became_peered":"2024-04-15T15:15:10.301882+0000","last_unstale":"2024-04-15T15:15:15.866222+0000","last_undegraded":"2024-04-15T15:15:15.866222+0000","last_fullsized":"2024-04-15T15:15:15.866222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4499,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":104773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":104807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":695653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:27.672 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:15:27.673 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T15:15:27.673 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-15T15:15:27.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-15T15:15:27.673 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-15T15:15:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.809+0000 7fa63d8a0640 1 Processor -- start 2024-04-15T15:15:27.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.809+0000 7fa63d8a0640 1 -- start start 2024-04-15T15:15:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa63d8a0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa6380d2360 0x7fa6380d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa63d8a0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa63815eae0 0x7fa6380d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa63d8a0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6380b47a0 con 0x7fa63815e700 2024-04-15T15:15:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa63d8a0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa6380b53b0 con 0x7fa63815eae0 2024-04-15T15:15:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa63d8a0640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa6380b4a80 con 0x7fa6380d2360 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa63815eae0 0x7fa6380d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa63815eae0 0x7fa6380d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36492/0 (socket says 172.21.15.70:36492) 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 -- 172.21.15.70:0/3267602505 learned_addr learned my addr 172.21.15.70:0/3267602505 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa636ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa63815e700 0x7fa6380b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44694/0 (socket says 172.21.15.70:44694) 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa62e7fc640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa6380d2360 0x7fa6380d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 -- 172.21.15.70:0/3267602505 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa6380d2360 msgr2=0x7fa6380d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa6380d2360 0x7fa6380d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 -- 172.21.15.70:0/3267602505 >> v1:172.21.15.70:6789/0 conn(0x7fa63815e700 legacy=0x7fa6380b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6377fe640 1 -- 172.21.15.70:0/3267602505 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6380b4d60 con 0x7fa63815eae0 2024-04-15T15:15:27.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.810+0000 7fa6367fc640 1 -- 172.21.15.70:0/3267602505 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1114361009 0 0) 0x7fa6380b47a0 con 0x7fa63815e700 2024-04-15T15:15:27.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa6377fe640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815eae0 0x7fa6380d30a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e50 tx=0x7fa63002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62c4b5bf361385c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa6367fc640 1 -- 172.21.15.70:0/3267602505 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63002cb00 con 0x7fa63815eae0 2024-04-15T15:15:27.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa6367fc640 1 -- 172.21.15.70:0/3267602505 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63002cca0 con 0x7fa63815eae0 2024-04-15T15:15:27.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa6367fc640 1 -- 172.21.15.70:0/3267602505 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630034610 con 0x7fa63815eae0 2024-04-15T15:15:27.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 -- 172.21.15.70:0/3267602505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815eae0 msgr2=0x7fa6380d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815eae0 0x7fa6380d30a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa630009e50 tx=0x7fa63002bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 -- 172.21.15.70:0/3267602505 shutdown_connections 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815eae0 0x7fa6380d30a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/3267602505 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa6380d2360 0x7fa6380d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.811+0000 7fa63d8a0640 1 -- 172.21.15.70:0/3267602505 >> 172.21.15.70:0/3267602505 conn(0x7fa63808bf60 msgr2=0x7fa6380d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:27.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.812+0000 7fa63d8a0640 1 -- 172.21.15.70:0/3267602505 shutdown_connections 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.812+0000 7fa63d8a0640 1 -- 172.21.15.70:0/3267602505 wait complete. 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.813+0000 7fa63d8a0640 1 Processor -- start 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.813+0000 7fa63d8a0640 1 -- start start 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6380d2360 0x7fa638161990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa63815eae0 0x7fa6380b8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380d1d40 con 0x7fa63815e700 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa6380b4450 con 0x7fa6380d2360 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa6380d1ec0 con 0x7fa63815eae0 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36496/0 (socket says 172.21.15.70:36496) 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 -- 172.21.15.70:0/1135071401 learned_addr learned my addr 172.21.15.70:0/1135071401 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa636ffd640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6380d2360 0x7fa638161990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa6377fe640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa63815eae0 0x7fa6380b8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 -- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa63815eae0 msgr2=0x7fa6380b8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa63815eae0 0x7fa6380b8370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 -- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6380d2360 msgr2=0x7fa638161990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:27.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6380d2360 0x7fa638161990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:27.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 -- 172.21.15.70:0/1135071401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6381608f0 con 0x7fa63815e700 2024-04-15T15:15:27.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa62e7fc640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d800 tx=0x7fa62000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000b660 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa620019070 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000ba20 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa638160bd0 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.814+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa638161190 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.815+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6380689e0 con 0x7fa63815e700 2024-04-15T15:15:27.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.816+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa62000bbc0 con 0x7fa63815e700 2024-04-15T15:15:27.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.816+0000 7fa634ff9640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 0x7fa61806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:27.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.816+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa62008c2c0 con 0x7fa63815e700 2024-04-15T15:15:27.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.816+0000 7fa636ffd640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 0x7fa61806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:27.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.817+0000 7fa636ffd640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 0x7fa61806edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa628001020 tx=0x7fa628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:27.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:27.821+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa62001c360 con 0x7fa63815e700 2024-04-15T15:15:28.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.042+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa63815b110 con 0x7fa63815e700 2024-04-15T15:15:28.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.045+0000 7fa634ff9640 1 -- 172.21.15.70:0/1135071401 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+174135 (secure 0 0 0) 0x7fa62005a3b0 con 0x7fa63815e700 2024-04-15T15:15:28.049 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.047+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 msgr2=0x7fa61806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.047+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 0x7fa61806edb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa628001020 tx=0x7fa628009040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 msgr2=0x7fa638155ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa62000d800 tx=0x7fa62000dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 shutdown_connections 2024-04-15T15:15:28.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa63815eae0 0x7fa6380b8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa61806c8f0 0x7fa61806edb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa63815e700 0x7fa638155ac0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 --2- 172.21.15.70:0/1135071401 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6380d2360 0x7fa638161990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 >> 172.21.15.70:0/1135071401 conn(0x7fa63808bf60 msgr2=0x7fa6380d0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 shutdown_connections 2024-04-15T15:15:28.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.048+0000 7fa63d8a0640 1 -- 172.21.15.70:0/1135071401 wait complete. 2024-04-15T15:15:28.060 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2172332499},{"type":"v1","addr":"172.21.15.70:6833","nonce":2172332499}]},"active_addr":"172.21.15.70:6833/2172332499","active_change":"2024-04-15T15:14:50.059974+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4102,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":619755557}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":3505995421}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":256530381}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.70:0","nonce":2064979379}]}]} 2024-04-15T15:15:28.064 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-15T15:15:28.064 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-15T15:15:28.064 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:28.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.167+0000 7fbb5480d640 1 Processor -- start 2024-04-15T15:15:28.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.167+0000 7fbb5480d640 1 -- start start 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbb5015e700 0x7fbb500b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb500d2360 0x7fbb500d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb500d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb500b53b0 con 0x7fbb500d2360 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbb500b47a0 con 0x7fbb5015e700 2024-04-15T15:15:28.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb5480d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbb500b4a80 con 0x7fbb5015eae0 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb45d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb500d2360 0x7fbb500d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb45d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb500d2360 0x7fbb500d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36504/0 (socket says 172.21.15.70:36504) 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb45d9b640 1 -- 172.21.15.70:0/3775569159 learned_addr learned my addr 172.21.15.70:0/3775569159 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb4e59c640 1 --2- 172.21.15.70:0/3775569159 >> v2:172.21.15.96:3300/0 conn(0x7fbb5015e700 0x7fbb500b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.168+0000 7fbb4ed9d640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb500d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 -- 172.21.15.70:0/3775569159 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 msgr2=0x7fbb500d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb500d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 -- 172.21.15.70:0/3775569159 >> v2:172.21.15.96:3300/0 conn(0x7fbb5015e700 msgr2=0x7fbb500b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/3775569159 >> v2:172.21.15.96:3300/0 conn(0x7fbb5015e700 0x7fbb500b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 -- 172.21.15.70:0/3775569159 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb500b4d60 con 0x7fbb500d2360 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb500d2b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb5006c6c0 tx=0x7fbb3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2a37e2d2cfbe66 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.169+0000 7fbb4dd9b640 1 -- 172.21.15.70:0/3775569159 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38035020 con 0x7fbb500d2360 2024-04-15T15:15:28.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb4dd9b640 1 -- 172.21.15.70:0/3775569159 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb38031870 con 0x7fbb500d2360 2024-04-15T15:15:28.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb4dd9b640 1 -- 172.21.15.70:0/3775569159 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb38031c40 con 0x7fbb500d2360 2024-04-15T15:15:28.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 -- 172.21.15.70:0/3775569159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 msgr2=0x7fbb500d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb500d2b60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbb5006c6c0 tx=0x7fbb3802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 -- 172.21.15.70:0/3775569159 shutdown_connections 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb500d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 --2- 172.21.15.70:0/3775569159 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb500d2b60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 --2- 172.21.15.70:0/3775569159 >> v2:172.21.15.96:3300/0 conn(0x7fbb5015e700 0x7fbb500b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 -- 172.21.15.70:0/3775569159 >> 172.21.15.70:0/3775569159 conn(0x7fbb5008bf60 msgr2=0x7fbb500d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 -- 172.21.15.70:0/3775569159 shutdown_connections 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.170+0000 7fbb5480d640 1 -- 172.21.15.70:0/3775569159 wait complete. 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 Processor -- start 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 -- start start 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb5015e700 0x7fbb50154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb50160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb500b5d80 con 0x7fbb500d2360 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb500b4450 con 0x7fbb5015e700 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb5480d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbb500d1d40 con 0x7fbb5015eae0 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb4e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb4e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36506/0 (socket says 172.21.15.70:36506) 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.172+0000 7fbb4e59c640 1 -- 172.21.15.70:0/1264888645 learned_addr learned my addr 172.21.15.70:0/1264888645 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb5015e700 0x7fbb50154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4ed9d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb50160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 -- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 msgr2=0x7fbb50160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb50160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 -- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb5015e700 msgr2=0x7fbb50154ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb5015e700 0x7fbb50154ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 -- 172.21.15.70:0/1264888645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb501577e0 con 0x7fbb500d2360 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb4e59c640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000d830 tx=0x7fbb4000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb400096a0 con 0x7fbb500d2360 2024-04-15T15:15:28.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb40019070 con 0x7fbb500d2360 2024-04-15T15:15:28.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb50157ac0 con 0x7fbb500d2360 2024-04-15T15:15:28.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40009a60 con 0x7fbb500d2360 2024-04-15T15:15:28.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.173+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb50155010 con 0x7fbb500d2360 2024-04-15T15:15:28.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.174+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb500689e0 con 0x7fbb500d2360 2024-04-15T15:15:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.175+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbb4001e460 con 0x7fbb500d2360 2024-04-15T15:15:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.175+0000 7fbb477fe640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 0x7fbb3006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.175+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbb4008c540 con 0x7fbb500d2360 2024-04-15T15:15:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.175+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 0x7fbb3006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.175+0000 7fbb45d9b640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 0x7fbb3006edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb38036000 tx=0x7fbb3803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.179+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb4005a630 con 0x7fbb500d2360 2024-04-15T15:15:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.356+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbb500ba500 con 0x7fbb500d2360 2024-04-15T15:15:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.357+0000 7fbb477fe640 1 -- 172.21.15.70:0/1264888645 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7fbb40059f30 con 0x7fbb500d2360 2024-04-15T15:15:28.359 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:28.359 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 msgr2=0x7fbb3006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 0x7fbb3006edb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbb38036000 tx=0x7fbb3803c040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 msgr2=0x7fbb5014a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000d830 tx=0x7fbb4000dcf0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 shutdown_connections 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb5015eae0 0x7fbb50160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb5015e700 0x7fbb50154ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbb3006c8f0 0x7fbb3006edb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 --2- 172.21.15.70:0/1264888645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb500d2360 0x7fbb5014a8f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 >> 172.21.15.70:0/1264888645 conn(0x7fbb5008bf60 msgr2=0x7fbb500d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 shutdown_connections 2024-04-15T15:15:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.359+0000 7fbb5480d640 1 -- 172.21.15.70:0/1264888645 wait complete. 2024-04-15T15:15:28.372 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-15T15:15:28.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:28.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.473+0000 7f0e618f8640 1 Processor -- start 2024-04-15T15:15:28.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.473+0000 7f0e618f8640 1 -- start start 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e618f8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0e5c15c690 0x7f0e5c06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e618f8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0e5c0cd580 0x7f0e5c0b5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e618f8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e5c067a60 con 0x7f0e5c06ff60 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e618f8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0e5c160ea0 con 0x7f0e5c15c690 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e618f8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0e5c160290 con 0x7f0e5c0cd580 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e5affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0e5c15c690 0x7f0e5c06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e5affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0e5c15c690 0x7f0e5c06fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36520/0 (socket says 172.21.15.70:36520) 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.474+0000 7f0e5affd640 1 -- 172.21.15.70:0/1139062424 learned_addr learned my addr 172.21.15.70:0/1139062424 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5a7fc640 1 --2- 172.21.15.70:0/1139062424 >> v2:172.21.15.96:3300/0 conn(0x7f0e5c0cd580 0x7f0e5c0b5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5affd640 1 -- 172.21.15.70:0/1139062424 >> v2:172.21.15.96:3300/0 conn(0x7f0e5c0cd580 msgr2=0x7f0e5c0b5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5affd640 1 --2- 172.21.15.70:0/1139062424 >> v2:172.21.15.96:3300/0 conn(0x7f0e5c0cd580 0x7f0e5c0b5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5affd640 1 -- 172.21.15.70:0/1139062424 >> v1:172.21.15.96:6789/0 conn(0x7f0e5c06ff60 legacy=0x7f0e5c06e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5affd640 1 -- 172.21.15.70:0/1139062424 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c160570 con 0x7f0e5c15c690 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.475+0000 7f0e5affd640 1 --2- 172.21.15.70:0/1139062424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c15c690 0x7f0e5c06fa20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c071b20 tx=0x7f0e4c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14c42a7a62ab427 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e59ffb640 1 -- 172.21.15.70:0/1139062424 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c02c9f0 con 0x7f0e5c15c690 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e59ffb640 1 -- 172.21.15.70:0/1139062424 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4c02cb90 con 0x7f0e5c15c690 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e59ffb640 1 -- 172.21.15.70:0/1139062424 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0343f0 con 0x7f0e5c15c690 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 -- 172.21.15.70:0/1139062424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c15c690 msgr2=0x7f0e5c06fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 --2- 172.21.15.70:0/1139062424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c15c690 0x7f0e5c06fa20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0e5c071b20 tx=0x7f0e4c02be50 comp rx=0 tx=0).stop 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 -- 172.21.15.70:0/1139062424 shutdown_connections 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 --2- 172.21.15.70:0/1139062424 >> v2:172.21.15.96:3300/0 conn(0x7f0e5c0cd580 0x7f0e5c0b5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 --2- 172.21.15.70:0/1139062424 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c15c690 0x7f0e5c06fa20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 -- 172.21.15.70:0/1139062424 >> 172.21.15.70:0/1139062424 conn(0x7f0e5c073360 msgr2=0x7f0e5c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 -- 172.21.15.70:0/1139062424 shutdown_connections 2024-04-15T15:15:28.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.476+0000 7f0e618f8640 1 -- 172.21.15.70:0/1139062424 wait complete. 2024-04-15T15:15:28.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.477+0000 7f0e618f8640 1 Processor -- start 2024-04-15T15:15:28.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 -- start start 2024-04-15T15:15:28.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 0x7f0e5c153580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 0x7f0e5c0b3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c0b63f0 con 0x7f0e5c0cd580 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c160b30 con 0x7f0e5c15c690 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e618f8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0e5c0b6570 con 0x7f0e5c06ff60 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.478+0000 7f0e5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36526/0 (socket says 172.21.15.70:36526) 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 -- 172.21.15.70:0/710220793 learned_addr learned my addr 172.21.15.70:0/710220793 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5affd640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 0x7f0e5c153580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5b7fe640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 0x7f0e5c0b3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 -- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 msgr2=0x7f0e5c153580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 0x7f0e5c153580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 -- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 msgr2=0x7f0e5c0b3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 0x7f0e5c0b3ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 -- 172.21.15.70:0/710220793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c154430 con 0x7f0e5c0cd580 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5b7fe640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 0x7f0e5c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5affd640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 0x7f0e5c153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e5a7fc640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e9b0 tx=0x7f0e4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44017070 con 0x7f0e5c0cd580 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e4400c9a0 con 0x7f0e5c0cd580 2024-04-15T15:15:28.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e4400cd70 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c153e40 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.479+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c1f4630 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.480+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5c0690f0 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.481+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0e4401a070 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.481+0000 7f0e53fff640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 0x7f0e3c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.481+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0e4408be90 con 0x7f0e5c0cd580 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.481+0000 7f0e5affd640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 0x7f0e3c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.481+0000 7f0e5affd640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 0x7f0e3c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c035000 tx=0x7f0e4c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.485+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0e44059f80 con 0x7f0e5c0cd580 2024-04-15T15:15:28.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.661+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e5c15ecd0 con 0x7f0e5c0cd580 2024-04-15T15:15:28.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.662+0000 7f0e53fff640 1 -- 172.21.15.70:0/710220793 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f0e44059880 con 0x7f0e5c0cd580 2024-04-15T15:15:28.664 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:28.664 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 msgr2=0x7f0e3c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 0x7f0e3c06edb0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c035000 tx=0x7f0e4c030000 comp rx=0 tx=0).stop 2024-04-15T15:15:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 msgr2=0x7f0e5c1501a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400e9b0 tx=0x7f0e4400ee70 comp rx=0 tx=0).stop 2024-04-15T15:15:28.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 shutdown_connections 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0e5c15c690 0x7f0e5c0b3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0e3c06c8f0 0x7f0e3c06edb0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0e5c0cd580 0x7f0e5c1501a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 --2- 172.21.15.70:0/710220793 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0e5c06ff60 0x7f0e5c153580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 >> 172.21.15.70:0/710220793 conn(0x7f0e5c073360 msgr2=0x7f0e5c0cda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 shutdown_connections 2024-04-15T15:15:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.664+0000 7f0e618f8640 1 -- 172.21.15.70:0/710220793 wait complete. 2024-04-15T15:15:28.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-15T15:15:28.679 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-15T15:15:28.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-15T15:15:28.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-15T15:15:28.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-15T15:15:28.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-15T15:15:28.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-15T15:15:28.681 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.851+0000 7f345c1ba640 1 Processor -- start 2024-04-15T15:15:28.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.852+0000 7f345c1ba640 1 -- start start 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f345c1ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 0x7f345406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f345c1ba640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f345415db40 con 0x7f3454071d10 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f345c1ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3454072170 con 0x7f3454072d80 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f345c1ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f345415cfc0 con 0x7f34540d0da0 2024-04-15T15:15:28.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f3459755640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3454071d10 0x7f34540d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44704/0 (socket says 172.21.15.70:44704) 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f3459755640 1 -- 172.21.15.70:0/3552812336 learned_addr learned my addr 172.21.15.70:0/3552812336 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.853+0000 7f345a757640 1 --2- 172.21.15.70:0/3552812336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 0x7f345406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4139127096 0 0) 0x7f345415db40 con 0x7f3454071d10 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3440003520 con 0x7f3454071d10 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2095388472 0 0) 0x7f3454072170 con 0x7f3454072d80 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f345415db40 con 0x7f3454072d80 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f345a757640 1 -- 172.21.15.70:0/3552812336 >> v1:172.21.15.96:6789/0 conn(0x7f3454072d80 legacy=0x7f34540af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f345a757640 1 -- 172.21.15.70:0/3552812336 >> v1:172.21.15.70:6789/0 conn(0x7f3454071d10 legacy=0x7f34540d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.854+0000 7f345a757640 1 -- 172.21.15.70:0/3552812336 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345415d2a0 con 0x7f34540d0da0 2024-04-15T15:15:28.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.855+0000 7f5859334640 1 Processor -- start 2024-04-15T15:15:28.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.855+0000 7f345a757640 1 --2- 172.21.15.70:0/3552812336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 0x7f345406e060 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f344c009dc0 tx=0x7f344c02be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=155c2405531191a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7faad56f8640 1 Processor -- start 2024-04-15T15:15:28.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7faad56f8640 1 -- start start 2024-04-15T15:15:28.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faad56f8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faad00d07c0 0x7faad00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.859 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faad56f8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faad015b7d0 0x7faad00d0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c039070 con 0x7f34540d0da0 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f344c02cc40 con 0x7f34540d0da0 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7f3458f54640 1 -- 172.21.15.70:0/3552812336 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344c034440 con 0x7f34540d0da0 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 -- 172.21.15.70:0/3552812336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 msgr2=0x7f345406e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 --2- 172.21.15.70:0/3552812336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 0x7f345406e060 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f344c009dc0 tx=0x7f344c02be90 comp rx=0 tx=0).stop 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 -- 172.21.15.70:0/3552812336 shutdown_connections 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 --2- 172.21.15.70:0/3552812336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34540d0da0 0x7f345406e060 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 -- 172.21.15.70:0/3552812336 >> 172.21.15.70:0/3552812336 conn(0x7f345406be00 msgr2=0x7f345406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 -- 172.21.15.70:0/3552812336 shutdown_connections 2024-04-15T15:15:28.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.855+0000 7f9dee674640 1 Processor -- start 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.856+0000 7f9dee674640 1 -- start start 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9dee674640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9de80b3160 0x7f9de80b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9dee674640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de80b3aa0 con 0x7f9de81632f0 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9dee674640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de81664d0 con 0x7f9de815d160 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9dee674640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9de8164ac0 con 0x7f9de80b3160 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9de77fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9de81632f0 0x7f9de8165dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44722/0 (socket says 172.21.15.70:44722) 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9de77fe640 1 -- 172.21.15.70:0/84969724 learned_addr learned my addr 172.21.15.70:0/84969724 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f9decc11640 1 --2- 172.21.15.70:0/84969724 >> v2:172.21.15.96:3300/0 conn(0x7f9de80b3160 0x7f9de80b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faad56f8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faad00d10e0 con 0x7faad015bf10 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faad56f8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7faad00b4660 con 0x7faad015b7d0 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faad56f8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faad00b3a50 con 0x7faad00d07c0 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faacf7fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7faad015bf10 0x7faad00b4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55892/0 (socket says 172.21.15.70:55892) 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faacf7fe640 1 -- 172.21.15.70:0/3208662465 learned_addr learned my addr 172.21.15.70:0/3208662465 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7faaceffd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faad00d07c0 0x7faad00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faacdffb640 1 -- 172.21.15.70:0/3208662465 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 621851403 0 0) 0x7faad00d10e0 con 0x7faad015bf10 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.855+0000 7f5859334640 1 -- start start 2024-04-15T15:15:28.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5859334640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 0x7f584c03c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5859334640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 0x7f584c053d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5859334640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f584c058980 con 0x7f584c0e3040 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5859334640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f584c006860 con 0x7f584c0e78d0 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5859334640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f584c058c60 con 0x7f584c0e3420 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5853fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f584c0e3040 0x7f584c058130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44710/0 (socket says 172.21.15.70:44710) 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f5853fff640 1 -- 172.21.15.70:0/4062033107 learned_addr learned my addr 172.21.15.70:0/4062033107 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f58537fe640 1 --2- 172.21.15.70:0/4062033107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 0x7f584c03c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f5858b33640 1 --2- 172.21.15.70:0/4062033107 >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 0x7f584c053d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523983499 0 0) 0x7f584c058980 con 0x7f584c0e3040 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5844003520 con 0x7f584c0e3040 2024-04-15T15:15:28.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faacdffb640 1 -- 172.21.15.70:0/3208662465 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab8003520 con 0x7faad015bf10 2024-04-15T15:15:28.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.857+0000 7f345c1ba640 1 -- 172.21.15.70:0/3552812336 wait complete. 2024-04-15T15:15:28.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7fec074ce640 1 Processor -- start 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faaceffd640 1 -- 172.21.15.70:0/3208662465 >> v2:172.21.15.70:3300/0 conn(0x7faad015b7d0 msgr2=0x7faad00d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faaceffd640 1 --2- 172.21.15.70:0/3208662465 >> v2:172.21.15.70:3300/0 conn(0x7faad015b7d0 0x7faad00d0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faaceffd640 1 -- 172.21.15.70:0/3208662465 >> v1:172.21.15.96:6789/0 conn(0x7faad015bf10 legacy=0x7faad00b4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7faaceffd640 1 -- 172.21.15.70:0/3208662465 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad00b3d30 con 0x7faad00d07c0 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f9decc11640 1 -- 172.21.15.70:0/84969724 >> v1:172.21.15.96:6789/0 conn(0x7f9de815d160 legacy=0x7f9de81643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f9decc11640 1 -- 172.21.15.70:0/84969724 >> v1:172.21.15.70:6789/0 conn(0x7f9de81632f0 legacy=0x7f9de8165dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f9decc11640 1 -- 172.21.15.70:0/84969724 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de8164d20 con 0x7f9de80b3160 2024-04-15T15:15:28.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f345c1ba640 1 Processor -- start 2024-04-15T15:15:28.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f345c1ba640 1 -- start start 2024-04-15T15:15:28.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f33ef98d640 1 Processor -- start 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f9decc11640 1 --2- 172.21.15.70:0/84969724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de80b3560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ffb4916950aaa94 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f58537fe640 1 -- 172.21.15.70:0/4062033107 >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 msgr2=0x7f584c053d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2304773605 0 0) 0x7f5844003520 con 0x7f584c0e3040 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f58537fe640 1 --2- 172.21.15.70:0/4062033107 >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 0x7f584c053d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f58537fe640 1 -- 172.21.15.70:0/4062033107 >> v1:172.21.15.70:6789/0 conn(0x7f584c0e3040 legacy=0x7f584c058130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.858+0000 7f58537fe640 1 -- 172.21.15.70:0/4062033107 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c006210 con 0x7f584c0e3420 2024-04-15T15:15:28.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f5858b33640 1 --2- 172.21.15.70:0/4062033107 >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 0x7f584c053d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:28.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7f33ef98d640 1 -- start start 2024-04-15T15:15:28.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7faaceffd640 1 --2- 172.21.15.70:0/3208662465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad00d07c0 0x7faad00b3470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faad006ee50 tx=0x7faac002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcafec9c90f18b1f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f58537fe640 1 --2- 172.21.15.70:0/4062033107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 0x7f584c03c180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f584c0069e0 tx=0x7f584002b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db85550d2f13f011 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f9de6ffd640 1 -- 172.21.15.70:0/84969724 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8037070 con 0x7f9de80b3160 2024-04-15T15:15:28.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f7117350640 1 Processor -- start 2024-04-15T15:15:28.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f7117350640 1 -- start start 2024-04-15T15:15:28.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f9de6ffd640 1 -- 172.21.15.70:0/84969724 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dd8030710 con 0x7f9de80b3160 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f9de6ffd640 1 -- 172.21.15.70:0/84969724 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd8030ae0 con 0x7f9de80b3160 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.859+0000 7fec074ce640 1 -- start start 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec074ce640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fec001552f0 0x7fec000cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec074ce640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fec0006fca0 0x7fec0015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec074ce640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec000cdb10 con 0x7fec0015e730 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec074ce640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fec000cfb80 con 0x7fec0006fca0 2024-04-15T15:15:28.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec074ce640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fec0006e800 con 0x7fec001552f0 2024-04-15T15:15:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7faacdffb640 1 -- 172.21.15.70:0/3208662465 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0034070 con 0x7faad00d07c0 2024-04-15T15:15:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7faacdffb640 1 -- 172.21.15.70:0/3208662465 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac0030840 con 0x7faad00d07c0 2024-04-15T15:15:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7faacdffb640 1 -- 172.21.15.70:0/3208662465 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac0030c50 con 0x7faad00d07c0 2024-04-15T15:15:28.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec0526a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fec001552f0 0x7fec000cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec0526a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fec001552f0 0x7fec000cd5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60740/0 (socket says 172.21.15.70:60740) 2024-04-15T15:15:28.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7fec0526a640 1 -- 172.21.15.70:0/1221186753 learned_addr learned my addr 172.21.15.70:0/1221186753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33ef98d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e0050040 0x7f33e00542e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7febf7fff640 1 -- 172.21.15.70:0/1221186753 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 826703336 0 0) 0x7fec000cdb10 con 0x7fec0015e730 2024-04-15T15:15:28.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7febf7fff640 1 -- 172.21.15.70:0/1221186753 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febec003520 con 0x7fec0015e730 2024-04-15T15:15:28.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 -- 172.21.15.70:0/3208662465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad00d07c0 msgr2=0x7faad00b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 --2- 172.21.15.70:0/3208662465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad00d07c0 0x7faad00b3470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faad006ee50 tx=0x7faac002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:28.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33ef98d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e0005f10 con 0x7f33e0053e50 2024-04-15T15:15:28.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33ef98d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33e0053280 con 0x7f33e0048100 2024-04-15T15:15:28.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 -- 172.21.15.70:0/3208662465 shutdown_connections 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 --2- 172.21.15.70:0/3208662465 >> v2:172.21.15.70:3300/0 conn(0x7faad015b7d0 0x7faad00d0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 --2- 172.21.15.70:0/3208662465 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad00d07c0 0x7faad00b3470 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 -- 172.21.15.70:0/3208662465 >> 172.21.15.70:0/3208662465 conn(0x7faad006be00 msgr2=0x7faad006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f9dee674640 1 -- 172.21.15.70:0/84969724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 msgr2=0x7f9de80b3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f9dee674640 1 --2- 172.21.15.70:0/84969724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de80b3560 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9dd80097a0 tx=0x7f9dd802b680 comp rx=0 tx=0).stop 2024-04-15T15:15:28.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33ef98d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f33e00e7b90 con 0x7f33e0050040 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7fec0526a640 1 -- 172.21.15.70:0/1221186753 >> v2:172.21.15.70:3300/0 conn(0x7fec0006fca0 msgr2=0x7fec0015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7fec0526a640 1 --2- 172.21.15.70:0/1221186753 >> v2:172.21.15.70:3300/0 conn(0x7fec0006fca0 0x7fec0015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7fec0526a640 1 -- 172.21.15.70:0/1221186753 >> v1:172.21.15.70:6789/0 conn(0x7fec0015e730 legacy=0x7fec0015b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7fec0526a640 1 -- 172.21.15.70:0/1221186753 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec000cfdd0 con 0x7fec001552f0 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33edf2a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e0050040 0x7f33e00542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7faad56f8640 1 -- 172.21.15.70:0/3208662465 shutdown_connections 2024-04-15T15:15:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33edf2a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e0050040 0x7f33e00542e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42802/0 (socket says 172.21.15.70:42802) 2024-04-15T15:15:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f9dee674640 1 -- 172.21.15.70:0/84969724 shutdown_connections 2024-04-15T15:15:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f9dee674640 1 --2- 172.21.15.70:0/84969724 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de80b3560 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f9dee674640 1 -- 172.21.15.70:0/84969724 >> 172.21.15.70:0/84969724 conn(0x7f9de80b5680 msgr2=0x7f9de80b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f33edf2a640 1 -- 172.21.15.70:0/1015276298 learned_addr learned my addr 172.21.15.70:0/1015276298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7faad56f8640 1 -- 172.21.15.70:0/3208662465 wait complete. 2024-04-15T15:15:28.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f7117350640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f711015d160 0x7f71101643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f7117350640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71100b3160 0x7f71100b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f7117350640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71101664d0 con 0x7f71101632f0 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f7117350640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f71100b3af0 con 0x7f711015d160 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f7117350640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7110164d20 con 0x7f71100b3160 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f71148eb640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f71101632f0 0x7f7110165dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55918/0 (socket says 172.21.15.70:55918) 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f71148eb640 1 -- 172.21.15.70:0/700284907 learned_addr learned my addr 172.21.15.70:0/700284907 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f71150ec640 1 --2- 172.21.15.70:0/700284907 >> v2:172.21.15.96:3300/0 conn(0x7f711015d160 0x7f71101643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 0x7f3454159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5840037070 con 0x7f584c0e3420 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5840030510 con 0x7f584c0e3420 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f5852ffd640 1 -- 172.21.15.70:0/4062033107 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848001670 con 0x7f584c0e3420 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3454072d80 0x7f3454099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34540d0da0 0x7f34541413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34540afaf0 con 0x7f34540d0da0 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f345415d860 con 0x7f3454071d10 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.860+0000 7f345c1ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f345415cb90 con 0x7f3454072d80 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 0x7f3454159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 0x7f3454159d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60748/0 (socket says 172.21.15.70:60748) 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 -- 172.21.15.70:0/1759535057 learned_addr learned my addr 172.21.15.70:0/1759535057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f345a757640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34540d0da0 0x7f34541413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459755640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3454072d80 0x7f3454099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 -- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3454072d80 msgr2=0x7f3454099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3454072d80 0x7f3454099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 -- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34540d0da0 msgr2=0x7f34541413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34540d0da0 0x7f34541413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.861+0000 7f3459f56640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3454099b30 con 0x7f3454071d10 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 135862033 0 0) 0x7f33e0005f10 con 0x7f33e0053e50 2024-04-15T15:15:28.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f33d4003520 con 0x7f33e0053e50 2024-04-15T15:15:28.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f71150ec640 1 -- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71100b3160 msgr2=0x7f71100b35b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:28.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f71150ec640 1 --2- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71100b3160 0x7f71100b35b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2364687388 0 0) 0x7f33d4003520 con 0x7f33e0053e50 2024-04-15T15:15:28.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e0050040 msgr2=0x7f33e00542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e0050040 0x7f33e00542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 >> v1:172.21.15.96:6789/0 conn(0x7f33e0048100 legacy=0x7f33e0006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e00061b0 con 0x7f33e0053e50 2024-04-15T15:15:28.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f33e4001670 con 0x7f33e0053e50 2024-04-15T15:15:28.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f464359e640 1 Processor -- start 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 >> v1:172.21.15.70:6789/0 conn(0x7f33e0053e50 legacy=0x7f33e004f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d40039e0 0x7f33d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d4004800 0x7f33d4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33d40051c0 0x7f33d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f464359e640 1 -- start start 2024-04-15T15:15:28.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33e00e7b90 con 0x7f33d40039e0 2024-04-15T15:15:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33d400bff0 con 0x7f33d4004800 2024-04-15T15:15:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f33d400c1f0 con 0x7f33d40051c0 2024-04-15T15:15:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f5859334640 1 -- 172.21.15.70:0/4062033107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 msgr2=0x7f584c03c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f5859334640 1 --2- 172.21.15.70:0/4062033107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 0x7f584c03c180 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f584c0069e0 tx=0x7f584002b710 comp rx=0 tx=0).stop 2024-04-15T15:15:28.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f3459f56640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 0x7f3454159d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f344400c500 tx=0x7f344400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f7107fff640 1 -- 172.21.15.70:0/700284907 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005964734 0 0) 0x7f71101664d0 con 0x7f71101632f0 2024-04-15T15:15:28.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7f71150ec640 1 -- 172.21.15.70:0/700284907 >> v1:172.21.15.96:6789/0 conn(0x7f71101632f0 legacy=0x7f7110165dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f71150ec640 1 -- 172.21.15.70:0/700284907 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7110165000 con 0x7f711015d160 2024-04-15T15:15:28.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33edf2a640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33d40051c0 0x7f33d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ed729640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d40039e0 0x7f33d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 -- 172.21.15.70:0/4062033107 shutdown_connections 2024-04-15T15:15:28.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 --2- 172.21.15.70:0/4062033107 >> v2:172.21.15.96:3300/0 conn(0x7f584c0e78d0 0x7f584c053d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 --2- 172.21.15.70:0/4062033107 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3420 0x7f584c03c180 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 -- 172.21.15.70:0/4062033107 >> 172.21.15.70:0/4062033107 conn(0x7f584c03aa40 msgr2=0x7f584c03ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d4004800 0x7f33d4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34440052d0 con 0x7f3454071d10 2024-04-15T15:15:28.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3444005490 con 0x7f3454071d10 2024-04-15T15:15:28.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444012870 con 0x7f3454071d10 2024-04-15T15:15:28.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 -- 172.21.15.70:0/4062033107 shutdown_connections 2024-04-15T15:15:28.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f5859334640 1 -- 172.21.15.70:0/4062033107 wait complete. 2024-04-15T15:15:28.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f345c1ba640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3454099dd0 con 0x7f3454071d10 2024-04-15T15:15:28.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f345c1ba640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f345415bcb0 con 0x7f3454071d10 2024-04-15T15:15:28.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.862+0000 7fec0526a640 1 --2- 172.21.15.70:0/1221186753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 0x7fec000cd5d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec00070120 tx=0x7febfc02df50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b42f453a463007e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7febf7fff640 1 -- 172.21.15.70:0/1221186753 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc034070 con 0x7fec001552f0 2024-04-15T15:15:28.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7febf7fff640 1 -- 172.21.15.70:0/1221186753 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc02ed00 con 0x7fec001552f0 2024-04-15T15:15:28.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7febf7fff640 1 -- 172.21.15.70:0/1221186753 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc038620 con 0x7fec001552f0 2024-04-15T15:15:28.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f9dee674640 1 -- 172.21.15.70:0/84969724 shutdown_connections 2024-04-15T15:15:28.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f464359e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c038620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 -- 172.21.15.70:0/1221186753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 msgr2=0x7fec000cd5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 --2- 172.21.15.70:0/1221186753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 0x7fec000cd5d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec00070120 tx=0x7febfc02df50 comp rx=0 tx=0).stop 2024-04-15T15:15:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f464359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c04f530 con 0x7f463c0e1080 2024-04-15T15:15:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7faad56f8640 1 Processor -- start 2024-04-15T15:15:28.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f464359e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c04f810 con 0x7f463c050dc0 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f464359e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f463c04faf0 con 0x7f463c03a5c0 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f4641d9b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f463c050dc0 0x7f463c005fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55930/0 (socket says 172.21.15.70:55930) 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f4641d9b640 1 -- 172.21.15.70:0/120524429 learned_addr learned my addr 172.21.15.70:0/120524429 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 -- 172.21.15.70:0/1221186753 shutdown_connections 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 --2- 172.21.15.70:0/1221186753 >> v2:172.21.15.70:3300/0 conn(0x7fec0006fca0 0x7fec0015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 --2- 172.21.15.70:0/1221186753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 0x7fec000cd5d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7fec074ce640 1 -- 172.21.15.70:0/1221186753 >> 172.21.15.70:0/1221186753 conn(0x7fec0006b9e0 msgr2=0x7fec0006bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 -- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33d40051c0 msgr2=0x7f33d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33d40051c0 0x7f33d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 -- 172.21.15.70:0/1015276298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d40039e0 msgr2=0x7f33d4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d40039e0 0x7f33d4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.863+0000 7f33ecf28640 1 -- 172.21.15.70:0/1015276298 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d4003520 con 0x7f33d4004800 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7faad56f8640 1 -- start start 2024-04-15T15:15:28.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f345c1ba640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f345415c090 con 0x7f3454071d10 2024-04-15T15:15:28.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f4642d9d640 1 --2- 172.21.15.70:0/120524429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c038620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.864+0000 7f9dee674640 1 -- 172.21.15.70:0/84969724 wait complete. 2024-04-15T15:15:28.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d4004800 0x7f33d4004c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f33e4007f50 tx=0x7f33d4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9cd4cbca3a065f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f5859334640 1 Processor -- start 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 413466385 0 0) 0x7f463c04f810 con 0x7f463c050dc0 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f462c003520 con 0x7f463c050dc0 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f4642d9d640 1 -- 172.21.15.70:0/120524429 >> v1:172.21.15.96:6789/0 conn(0x7f463c050dc0 legacy=0x7f463c005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 <== mon.1 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3999792115 0 0) 0x7f462c003520 con 0x7f463c050dc0 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f4642d9d640 1 -- 172.21.15.70:0/120524429 >> v1:172.21.15.70:6789/0 conn(0x7f463c0e1080 legacy=0x7f463c039eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f71150ec640 1 --2- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f71101643b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eda860b35c2b944d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f5859334640 1 -- start start 2024-04-15T15:15:28.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f7107fff640 1 -- 172.21.15.70:0/700284907 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c0315a0 con 0x7f711015d160 2024-04-15T15:15:28.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f7107fff640 1 -- 172.21.15.70:0/700284907 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c037070 con 0x7f711015d160 2024-04-15T15:15:28.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7107fff640 1 -- 172.21.15.70:0/700284907 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c031960 con 0x7f711015d160 2024-04-15T15:15:28.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faad56f8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 0x7faad00b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.865+0000 7f4642d9d640 1 -- 172.21.15.70:0/120524429 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c04fdd0 con 0x7f463c03a5c0 2024-04-15T15:15:28.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7fec074ce640 1 -- 172.21.15.70:0/1221186753 shutdown_connections 2024-04-15T15:15:28.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7fec074ce640 1 -- 172.21.15.70:0/1221186753 wait complete. 2024-04-15T15:15:28.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3444012a10 con 0x7f3454071d10 2024-04-15T15:15:28.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 0x7faad00b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f9dee674640 1 Processor -- start 2024-04-15T15:15:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faaceffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 0x7faad00b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36580/0 (socket says 172.21.15.70:36580) 2024-04-15T15:15:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faaceffd640 1 -- 172.21.15.70:0/3037870229 learned_addr learned my addr 172.21.15.70:0/3037870229 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f5859334640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 0x7f584c0de420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 -- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 msgr2=0x7f71101643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 --2- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f71101643b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c02b840 comp rx=0 tx=0).stop 2024-04-15T15:15:28.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faad56f8640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad015b7d0 0x7faad00ce200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 -- 172.21.15.70:0/700284907 shutdown_connections 2024-04-15T15:15:28.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f5859334640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f584c0e3420 0x7f584c0526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f344a7fc640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f343806c9c0 0x7f343806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7faad56f8640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faad015bf10 0x7faad014fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f5853fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 0x7f584c0de420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f5853fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 0x7f584c0de420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42828/0 (socket says 172.21.15.70:42828) 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f5853fff640 1 -- 172.21.15.70:0/1200884876 learned_addr learned my addr 172.21.15.70:0/1200884876 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5859334640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f584c0e78d0 0x7f584c0e55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f344408bc50 con 0x7f3454071d10 2024-04-15T15:15:28.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 --2- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71100b3160 0x7f71100b35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f58537fe640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f584c0e3420 0x7f584c0526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f344a7fc640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f3438076130 0x7f3438078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3438078c40 con 0x7f3438076130 2024-04-15T15:15:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f344408c040 con 0x7f3454071d10 2024-04-15T15:15:28.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f584c054450 con 0x7f584c0e3420 2024-04-15T15:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7faace7fc640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad015b7d0 0x7faad00ce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faad015bc60 con 0x7faad00d07c0 2024-04-15T15:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5858b33640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f584c0e78d0 0x7f584c0e55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faad00b42f0 con 0x7faad015b7d0 2024-04-15T15:15:28.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f584c0542d0 con 0x7f584c0e78d0 2024-04-15T15:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faad00afa60 con 0x7faad015bf10 2024-04-15T15:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f584c0545d0 con 0x7f584c0e3040 2024-04-15T15:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4039070 con 0x7f33d4004800 2024-04-15T15:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e402c970 con 0x7f33d4004800 2024-04-15T15:15:28.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f33dffff640 1 -- 172.21.15.70:0/1015276298 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4007b70 con 0x7f33d4004800 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 -- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d4004800 msgr2=0x7f33d4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 --2- 172.21.15.70:0/1015276298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33d4004800 0x7f33d4004c80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f33e4007f50 tx=0x7f33d4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33edf2a640 1 -- 172.21.15.70:0/1015276298 reap_dead start 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7faacf7fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faad015bf10 0x7faad014fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 -- 172.21.15.70:0/1015276298 shutdown_connections 2024-04-15T15:15:28.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 -- 172.21.15.70:0/1015276298 >> 172.21.15.70:0/1015276298 conn(0x7f33e0009fe0 msgr2=0x7f33e000a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 --2- 172.21.15.70:0/700284907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f71101643b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f4642d9d640 1 --2- 172.21.15.70:0/120524429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c038620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4644051700 tx=0x7f46440a6e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623d4a7fc2d8745e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 -- 172.21.15.70:0/1015276298 shutdown_connections 2024-04-15T15:15:28.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f3459755640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f343806c9c0 0x7f343806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46440b4070 con 0x7f463c03a5c0 2024-04-15T15:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f33ef98d640 1 -- 172.21.15.70:0/1015276298 wait complete. 2024-04-15T15:15:28.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f345a757640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f3438076130 0x7f3438078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46440a7a50 con 0x7f463c03a5c0 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f464159a640 1 -- 172.21.15.70:0/120524429 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638001670 con 0x7f463c03a5c0 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5853fff640 1 -- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f584c0e78d0 msgr2=0x7f584c0e55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f5853fff640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f584c0e78d0 0x7f584c0e55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5853fff640 1 -- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f584c0e3420 msgr2=0x7f584c0526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5853fff640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f584c0e3420 0x7f584c0526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5853fff640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f584c050c40 con 0x7f584c0e3040 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5858b33640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f584c0e78d0 0x7f584c0e55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f58537fe640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f584c0e3420 0x7f584c0526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f7117350640 1 -- 172.21.15.70:0/700284907 >> 172.21.15.70:0/700284907 conn(0x7f71100b5680 msgr2=0x7f71100b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f7117350640 1 -- 172.21.15.70:0/700284907 shutdown_connections 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.867+0000 7f7117350640 1 -- 172.21.15.70:0/700284907 wait complete. 2024-04-15T15:15:28.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7fec074ce640 1 Processor -- start 2024-04-15T15:15:28.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faacf7fe640 1 -- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad015b7d0 msgr2=0x7faad00ce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faacf7fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faad015b7d0 0x7faad00ce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faacf7fe640 1 -- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 msgr2=0x7faad00b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5853fff640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 0x7f584c0de420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f584800c730 tx=0x7f584800cbf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faacf7fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 0x7faad00b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faacf7fe640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad0155090 con 0x7faad015bf10 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848005ad0 con 0x7f584c0e3040 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5848005c70 con 0x7f584c0e3040 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5848007290 con 0x7f584c0e3040 2024-04-15T15:15:28.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f584c050f20 con 0x7f584c0e3040 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.866+0000 7f9dee674640 1 -- start start 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de815ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9de815d160 0x7f9de81ee680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de815ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9de7fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de815ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60816/0 (socket says 172.21.15.70:60816) 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9de815d160 0x7f9de81ee680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9de77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9de815d160 0x7f9de81ee680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36594/0 (socket says 172.21.15.70:36594) 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9de81632f0 0x7f9de81eebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81655c0 con 0x7f9de815d160 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9de81652e0 con 0x7f9de80b3160 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9dee674640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9de815d540 con 0x7f9de81632f0 2024-04-15T15:15:28.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f9de7fff640 1 -- 172.21.15.70:0/313985982 learned_addr learned my addr 172.21.15.70:0/313985982 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7faaceffd640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faad00d07c0 0x7faad00b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:28.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7faacf7fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faad015bf10 0x7faad014fce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faac8001660 tx=0x7faac8001b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f345a757640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f3438076130 0x7f3438078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f3459755640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f343806c9c0 0x7f343806ee80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34540734b0 tx=0x7f3450007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f464359e640 1 -- 172.21.15.70:0/120524429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 msgr2=0x7f463c038620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f464359e640 1 --2- 172.21.15.70:0/120524429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c038620 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4644051700 tx=0x7f46440a6e60 comp rx=0 tx=0).stop 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 -- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9de81632f0 msgr2=0x7f9de81eebc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9de81632f0 0x7f9de81eebc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 -- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9de815d160 msgr2=0x7f9de81ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9de815d160 0x7f9de81ee680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de80b0120 con 0x7f9de80b3160 2024-04-15T15:15:28.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.868+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f584c0514e0 con 0x7f584c0e3040 2024-04-15T15:15:28.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de7fff640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de815ac70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c7d0 tx=0x7f9de000cc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7fec074ce640 1 -- start start 2024-04-15T15:15:28.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8008270 con 0x7faad015bf10 2024-04-15T15:15:28.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac80064c0 con 0x7faad015bf10 2024-04-15T15:15:28.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f7117350640 1 Processor -- start 2024-04-15T15:15:28.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac8006890 con 0x7faad015bf10 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de0017070 con 0x7f9de80b3160 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9de0003da0 con 0x7f9de80b3160 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9de00052b0 con 0x7f9de80b3160 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de80b0400 con 0x7f9de80b3160 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f7117350640 1 -- start start 2024-04-15T15:15:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.869+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de80b09c0 con 0x7f9de80b3160 2024-04-15T15:15:28.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f33ef98d640 1 Processor -- start 2024-04-15T15:15:28.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f33ef98d640 1 -- start start 2024-04-15T15:15:28.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== osd.2 v2:172.21.15.70:6808/3050098076 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3438078c40 con 0x7f3438076130 2024-04-15T15:15:28.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7fec074ce640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 0x7fec001507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad0155370 con 0x7faad015bf10 2024-04-15T15:15:28.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faad0155930 con 0x7faad015bf10 2024-04-15T15:15:28.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f464359e640 1 -- 172.21.15.70:0/120524429 shutdown_connections 2024-04-15T15:15:28.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f464359e640 1 --2- 172.21.15.70:0/120524429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c038620 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f464359e640 1 -- 172.21.15.70:0/120524429 >> 172.21.15.70:0/120524429 conn(0x7f463c009fa0 msgr2=0x7f463c00a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 0x7f71100ae400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f711014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71101632f0 0x7f711014eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f711015d540 con 0x7f71100b3160 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f71101655c0 con 0x7f711015d160 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7f7117350640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f711015d6c0 con 0x7f71101632f0 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f71150ec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 0x7f71100ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f71150ec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 0x7f71100ae400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36600/0 (socket says 172.21.15.70:36600) 2024-04-15T15:15:28.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f71150ec640 1 -- 172.21.15.70:0/230262695 learned_addr learned my addr 172.21.15.70:0/230262695 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7fec074ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 0x7fec001556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.870+0000 7fec074ce640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fec0015e730 0x7fec0015a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec074ce640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fec0015f3b0 con 0x7fec0006fca0 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec074ce640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fec000d0390 con 0x7fec001552f0 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec074ce640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fec0015f530 con 0x7fec0015e730 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 0x7fec001507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 0x7fec001507e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36598/0 (socket says 172.21.15.70:36598) 2024-04-15T15:15:28.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 -- 172.21.15.70:0/869340547 learned_addr learned my addr 172.21.15.70:0/869340547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9de80689e0 con 0x7f9de80b3160 2024-04-15T15:15:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d4004800 0x7f33e00dce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 0x7f33e00364a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e00441d0 0x7f33e0044650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ecf28640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 0x7f33e00364a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ecf28640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 0x7f33e00364a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60826/0 (socket says 172.21.15.70:60826) 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ecf28640 1 -- 172.21.15.70:0/1050643413 learned_addr learned my addr 172.21.15.70:0/1050643413 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f33e0006770 con 0x7f33d4004800 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f33e0006ae0 con 0x7f33e0036020 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f33e0007680 con 0x7f33e00441d0 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f33edf2a640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e00441d0 0x7f33e0044650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f71148eb640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f711014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f71158ed640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71101632f0 0x7f711014eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f584c0052c0 con 0x7f584c0e3040 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 -- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e00441d0 msgr2=0x7f33e0044650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f33e00441d0 0x7f33e0044650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 -- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d4004800 msgr2=0x7f33e00dce50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f33d4004800 0x7f33e00dce50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e0044b90 con 0x7f33e0036020 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f464359e640 1 -- 172.21.15.70:0/120524429 shutdown_connections 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7f464359e640 1 -- 172.21.15.70:0/120524429 wait complete. 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 -- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71101632f0 msgr2=0x7f711014eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71101632f0 0x7f711014eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 -- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 msgr2=0x7f711014e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f711015d160 0x7f711014e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71101604f0 con 0x7f71100b3160 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5848007430 con 0x7f584c0e3040 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 -- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fec0015e730 msgr2=0x7fec0015a810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fec0015e730 0x7fec0015a810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 -- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 msgr2=0x7fec001556d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-15T15:15:28.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fec001552f0 0x7fec001556d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.871+0000 7fec0526a640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec00155fd0 con 0x7fec0006fca0 2024-04-15T15:15:28.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7fec0526a640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 0x7fec001507e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7febfc03a040 tx=0x7febfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f71150ec640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 0x7f71100ae400 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ecf28640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 0x7f33e00364a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f33e400c1d0 tx=0x7f33e0006a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c032840 con 0x7f71100b3160 2024-04-15T15:15:28.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f710c0329e0 con 0x7f71100b3160 2024-04-15T15:15:28.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f711009a4a0 con 0x7f71100b3160 2024-04-15T15:15:28.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f710c032dd0 con 0x7f71100b3160 2024-04-15T15:15:28.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc02eb00 con 0x7fec0006fca0 2024-04-15T15:15:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f5850ff9640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f583806c9c0 0x7f583806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febfc0415a0 con 0x7fec0006fca0 2024-04-15T15:15:28.930 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f464359e640 1 Processor -- start 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9de001a070 con 0x7f9de80b3160 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de4ff9640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9dd406c9c0 0x7f9dd406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9de008b500 con 0x7f9de80b3160 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de4ff9640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9dd40760c0 0x7f9dd4078500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9dd4078bd0 con 0x7f9dd40760c0 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9de0092050 con 0x7f9de80b3160 2024-04-15T15:15:28.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71100ae940 con 0x7f71100b3160 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e4039040 con 0x7f33e0036020 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33e4005270 con 0x7f33e0036020 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e0045ef0 con 0x7f33e0036020 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e0045130 con 0x7f33e0036020 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e402c970 con 0x7f33e0036020 2024-04-15T15:15:28.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f464359e640 1 -- start start 2024-04-15T15:15:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.873+0000 7f9de77fe640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9dd406c9c0 0x7f9dd406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f9de77fe640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9dd406c9c0 0x7f9dd406ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9de8204a90 tx=0x7f9ddc007e00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f584808bcf0 con 0x7f584c0e3040 2024-04-15T15:15:28.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.872+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faad00689e0 con 0x7faad015bf10 2024-04-15T15:15:28.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f464359e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c0dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f464359e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f463c050dc0 0x7f463c0db490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f5850ff9640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f5838076140 0x7f5838078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febfc041990 con 0x7fec0006fca0 2024-04-15T15:15:28.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5838078c50 con 0x7f5838076140 2024-04-15T15:15:28.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f584808e350 con 0x7f584c0e3040 2024-04-15T15:15:28.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f9decc11640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9dd40760c0 0x7f9dd4078500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec001562b0 con 0x7fec0006fca0 2024-04-15T15:15:28.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f463c0e1080 0x7f463c0e3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c0dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.874+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33e00052f0 con 0x7f33e0036020 2024-04-15T15:15:28.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faac8006a30 con 0x7faad015bf10 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec00156870 con 0x7fec0006fca0 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c0dbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42862/0 (socket says 172.21.15.70:42862) 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f4641d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f463c050dc0 0x7f463c0db490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f4641d9b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f463c050dc0 0x7f463c0db490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60828/0 (socket says 172.21.15.70:60828) 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464359e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0066e0 con 0x7f463c0e1080 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464359e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f463c050390 con 0x7f463c050dc0 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464359e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f463c006860 con 0x7f463c03a5c0 2024-04-15T15:15:28.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f58537fe640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f583806c9c0 0x7f583806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f464259c640 1 -- 172.21.15.70:0/1064522175 learned_addr learned my addr 172.21.15.70:0/1064522175 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:28.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f9decc11640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9dd40760c0 0x7f9dd4078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f5858b33640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f5838076140 0x7f5838078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f711009ab60 con 0x7f71100b3160 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faab77fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faabc06c9c0 0x7faabc06ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faac8094940 con 0x7faad015bf10 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faab77fe640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7faabc076130 0x7faabc078570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faac8019e50 con 0x7faabc076130 2024-04-15T15:15:28.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faac8094d30 con 0x7faad015bf10 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 -- 172.21.15.70:0/1064522175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f463c050dc0 msgr2=0x7f463c0db490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f463c050dc0 0x7f463c0db490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 -- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f463c0e1080 msgr2=0x7f463c0e3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f463c0e1080 0x7f463c0e3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c0e2c40 con 0x7f463c03a5c0 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f464259c640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c0dbc70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f463400c570 tx=0x7f463400ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec000690f0 con 0x7fec0006fca0 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.875+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f33e40037f0 con 0x7f33e0036020 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f33ddffb640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33cc06cb10 0x7f33cc06efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f33e40ad430 con 0x7f33e0036020 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f33ddffb640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7f33cc076210 0x7f33cc078650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33e4007630 con 0x7f33cc076210 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f33e40ad8c0 con 0x7f33e0036020 2024-04-15T15:15:28.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7faaceffd640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faabc06c9c0 0x7faabc06ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== osd.4 v2:172.21.15.96:6800/290497787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9dd4078bd0 con 0x7f9dd40760c0 2024-04-15T15:15:28.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634017070 con 0x7f463c03a5c0 2024-04-15T15:15:28.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4634005480 con 0x7f463c03a5c0 2024-04-15T15:15:28.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46340037f0 con 0x7f463c03a5c0 2024-04-15T15:15:28.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c0e2f20 con 0x7f463c03a5c0 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c0e34e0 con 0x7f463c03a5c0 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7faace7fc640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7faabc076130 0x7faabc078570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7faaceffd640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faabc06c9c0 0x7faabc06ee80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faac002b9c0 tx=0x7faac003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.876+0000 7f5858b33640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f5838076140 0x7f5838078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== osd.1 v2:172.21.15.70:6800/1494795690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5838078c50 con 0x7f5838076140 2024-04-15T15:15:28.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f58537fe640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f583806c9c0 0x7f583806ee80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5840034000 tx=0x7f5840008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.877+0000 7f33ed729640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33cc06cb10 0x7f33cc06efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.878+0000 7f33edf2a640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7f33cc076210 0x7f33cc078650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.878+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febfc049020 con 0x7fec0006fca0 2024-04-15T15:15:28.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4628001030 con 0x7f463c03a5c0 2024-04-15T15:15:28.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7febf5ffb640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe406c940 0x7febe406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.878+0000 7f33edf2a640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7f33cc076210 0x7f33cc078650 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== osd.7 v2:172.21.15.96:6824/1112579369 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33e4007630 con 0x7f33cc076210 2024-04-15T15:15:28.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7fec04a69640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe406c940 0x7febe406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f463401a070 con 0x7f463c03a5c0 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f710c03b3f0 con 0x7f71100b3160 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f7105ffb640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f70f406c8f0 0x7f70f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f710c02f030 con 0x7f71100b3160 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f7105ffb640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f70f4075ff0 0x7f70f4078430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.879+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f710c04bec0 con 0x7f70f4075ff0 2024-04-15T15:15:28.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f710c0b22a0 con 0x7f71100b3160 2024-04-15T15:15:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f71148eb640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f70f406c8f0 0x7f70f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.878+0000 7faace7fc640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7faabc076130 0x7faabc078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== osd.0 v2:172.21.15.70:6816/2850943089 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7faac8019e50 con 0x7faabc076130 2024-04-15T15:15:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7febfc0780e0 con 0x7fec0006fca0 2024-04-15T15:15:28.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7febf5ffb640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7febe4076040 0x7febe4078480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febfc039e90 con 0x7febe4076040 2024-04-15T15:15:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7febfc077d10 con 0x7fec0006fca0 2024-04-15T15:15:28.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f71158ed640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f70f4075ff0 0x7f70f4078430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f4632ffd640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f462406c9c0 0x7f462406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f463408b1f0 con 0x7f463c03a5c0 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f4632ffd640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7f4624076140 0x7f4624078580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4634010e50 con 0x7f4624076140 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.880+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4634092050 con 0x7f463c03a5c0 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7f4641d9b640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f462406c9c0 0x7f462406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7f33ed729640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33cc06cb10 0x7f33cc06efd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33d8004e00 tx=0x7f33d8007cc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7f71158ed640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f70f4075ff0 0x7f70f4078430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7f4642d9d640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7f4624076140 0x7f4624078580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7fec04a69640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe406c940 0x7febe406ee00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fec001f3480 tx=0x7febf8007c60 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.881+0000 7f71148eb640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f70f406c8f0 0x7f70f406edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f71101fc1f0 tx=0x7f71080014b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.882+0000 7f4641d9b640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f462406c9c0 0x7f462406ee80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f463c048a30 tx=0x7f46380016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.882+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== osd.5 v2:172.21.15.96:6808/660710685 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f710c04bec0 con 0x7f70f4075ff0 2024-04-15T15:15:28.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.882+0000 7fec05a6b640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7febe4076040 0x7febe4078480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:28.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.882+0000 7f4642d9d640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7f4624076140 0x7f4624078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.882+0000 7fec05a6b640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7febe4076040 0x7febe4078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:28.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.883+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== osd.3 v2:172.21.15.70:6824/1402304147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4634010e50 con 0x7f4624076140 2024-04-15T15:15:28.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.883+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== osd.6 v2:172.21.15.96:6816/2431924296 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7febfc039e90 con 0x7febe4076040 2024-04-15T15:15:28.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.899+0000 7f345c1ba640 1 -- 172.21.15.70:0/1759535057 --> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f34540689e0 con 0x7f3438076130 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.900+0000 7f344a7fc640 1 -- 172.21.15.70:0/1759535057 <== osd.2 v2:172.21.15.70:6808/3050098076 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f34540689e0 con 0x7f3438076130 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f3438076130 msgr2=0x7f3438078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f3438076130 0x7f3438078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f343806c9c0 msgr2=0x7f343806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f343806c9c0 0x7f343806ee80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f34540734b0 tx=0x7f3450007d30 comp rx=0 tx=0).stop 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 msgr2=0x7f3454159d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 --2- 172.21.15.70:0/1759535057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3454071d10 0x7f3454159d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f344400c500 tx=0x7f344400c9c0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f345a757640 1 -- 172.21.15.70:0/1759535057 reap_dead start 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 shutdown_connections 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.901+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 >> 172.21.15.70:0/1759535057 conn(0x7f345406be00 msgr2=0x7f345415ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.902+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 shutdown_connections 2024-04-15T15:15:28.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.902+0000 7f3437fff640 1 -- 172.21.15.70:0/1759535057 wait complete. 2024-04-15T15:15:28.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.907+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 --> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9de8072900 con 0x7f9dd40760c0 2024-04-15T15:15:28.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.907+0000 7f9de4ff9640 1 -- 172.21.15.70:0/313985982 <== osd.4 v2:172.21.15.96:6800/290497787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9de8072900 con 0x7f9dd40760c0 2024-04-15T15:15:28.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.908+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9dd40760c0 msgr2=0x7f9dd4078500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.908+0000 7f9dee674640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9dd40760c0 0x7f9dd4078500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.908+0000 7f5859334640 1 -- 172.21.15.70:0/1200884876 --> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f584c04c860 con 0x7f5838076140 2024-04-15T15:15:28.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f5850ff9640 1 -- 172.21.15.70:0/1200884876 <== osd.1 v2:172.21.15.70:6800/1494795690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f584c04c860 con 0x7f5838076140 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9dd406c9c0 msgr2=0x7f9dd406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f9dee674640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9dd406c9c0 0x7f9dd406ee80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9de8204a90 tx=0x7f9ddc007e00 comp rx=0 tx=0).stop 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 msgr2=0x7f9de815ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f9dee674640 1 --2- 172.21.15.70:0/313985982 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9de80b3160 0x7f9de815ac70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de000c7d0 tx=0x7f9de000cc90 comp rx=0 tx=0).stop 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.909+0000 7f9decc11640 1 -- 172.21.15.70:0/313985982 reap_dead start 2024-04-15T15:15:28.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 shutdown_connections 2024-04-15T15:15:28.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 >> 172.21.15.70:0/313985982 conn(0x7f9de80b5680 msgr2=0x7f9de806ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 shutdown_connections 2024-04-15T15:15:28.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f5838076140 msgr2=0x7f5838078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f583e7fc640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f5838076140 0x7f5838078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.910+0000 7f9dee674640 1 -- 172.21.15.70:0/313985982 wait complete. 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f583806c9c0 msgr2=0x7f583806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f583806c9c0 0x7f583806ee80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f5840034000 tx=0x7f5840008030 comp rx=0 tx=0).stop 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 msgr2=0x7f584c0de420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 --2- 172.21.15.70:0/1200884876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f584c0e3040 0x7f584c0de420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f584800c730 tx=0x7f584800cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f5858b33640 1 -- 172.21.15.70:0/1200884876 reap_dead start 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 shutdown_connections 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 >> 172.21.15.70:0/1200884876 conn(0x7f584c03aa40 msgr2=0x7f584c056ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.911+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 shutdown_connections 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.912+0000 7f583e7fc640 1 -- 172.21.15.70:0/1200884876 wait complete. 2024-04-15T15:15:28.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 --> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f33e0052850 con 0x7f33cc076210 2024-04-15T15:15:28.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 --> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f71100999f0 con 0x7f70f4075ff0 2024-04-15T15:15:28.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 --> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec000b6210 con 0x7febe4076040 2024-04-15T15:15:28.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7f33ddffb640 1 -- 172.21.15.70:0/1050643413 <== osd.7 v2:172.21.15.96:6824/1112579369 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f33e0052850 con 0x7f33cc076210 2024-04-15T15:15:28.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7f7105ffb640 1 -- 172.21.15.70:0/230262695 <== osd.5 v2:172.21.15.96:6808/660710685 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f71100999f0 con 0x7f70f4075ff0 2024-04-15T15:15:28.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.913+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 --> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faad015a660 con 0x7faabc076130 2024-04-15T15:15:28.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7febf5ffb640 1 -- 172.21.15.70:0/869340547 <== osd.6 v2:172.21.15.96:6816/2431924296 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fec000b6210 con 0x7febe4076040 2024-04-15T15:15:28.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7f33cc076210 msgr2=0x7f33cc078650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7f33cc076210 0x7f33cc078650 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f70f4075ff0 msgr2=0x7f70f4078430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f70f4075ff0 0x7f70f4078430 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33cc06cb10 msgr2=0x7f33cc06efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7faab77fe640 1 -- 172.21.15.70:0/3037870229 <== osd.0 v2:172.21.15.70:6816/2850943089 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faad015a660 con 0x7faabc076130 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33cc06cb10 0x7f33cc06efd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f33d8004e00 tx=0x7f33d8007cc0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 msgr2=0x7f33e00364a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 --2- 172.21.15.70:0/1050643413 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f33e0036020 0x7f33e00364a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f33e400c1d0 tx=0x7f33e0006a50 comp rx=0 tx=0).stop 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33edf2a640 1 -- 172.21.15.70:0/1050643413 reap_dead start 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 shutdown_connections 2024-04-15T15:15:28.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 >> 172.21.15.70:0/1050643413 conn(0x7f33e0009fe0 msgr2=0x7f33e004a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7febe4076040 msgr2=0x7febe4078480 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7febe4076040 0x7febe4078480 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f70f406c8f0 msgr2=0x7f70f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe406c940 msgr2=0x7febe406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe406c940 0x7febe406ee00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fec001f3480 tx=0x7febf8007c60 comp rx=0 tx=0).stop 2024-04-15T15:15:28.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f70f406c8f0 0x7f70f406edb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f71101fc1f0 tx=0x7f71080014b0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 msgr2=0x7f71100ae400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 msgr2=0x7fec001507e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec074ce640 1 --2- 172.21.15.70:0/869340547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fec0006fca0 0x7fec001507e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7febfc03a040 tx=0x7febfc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7fec05a6b640 1 -- 172.21.15.70:0/869340547 reap_dead start 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 shutdown_connections 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 >> 172.21.15.70:0/869340547 conn(0x7fec0006b9e0 msgr2=0x7fec000cf070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7faabc076130 msgr2=0x7faabc078570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7faabc076130 0x7faabc078570 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faabc06c9c0 msgr2=0x7faabc06ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faabc06c9c0 0x7faabc06ee80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7faac002b9c0 tx=0x7faac003b040 comp rx=0 tx=0).stop 2024-04-15T15:15:28.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 shutdown_connections 2024-04-15T15:15:28.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 shutdown_connections 2024-04-15T15:15:28.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f33ef98d640 1 -- 172.21.15.70:0/1050643413 wait complete. 2024-04-15T15:15:28.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7fec074ce640 1 -- 172.21.15.70:0/869340547 wait complete. 2024-04-15T15:15:28.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faad015bf10 msgr2=0x7faad014fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 --2- 172.21.15.70:0/3037870229 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faad015bf10 0x7faad014fce0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faac8001660 tx=0x7faac8001b20 comp rx=0 tx=0).stop 2024-04-15T15:15:28.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faacf7fe640 1 -- 172.21.15.70:0/3037870229 reap_dead start 2024-04-15T15:15:28.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 shutdown_connections 2024-04-15T15:15:28.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 >> 172.21.15.70:0/3037870229 conn(0x7faad006be00 msgr2=0x7faad01547c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 shutdown_connections 2024-04-15T15:15:28.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f7117350640 1 --2- 172.21.15.70:0/230262695 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71100b3160 0x7f71100ae400 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f710c0097a0 tx=0x7f710c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7faad56f8640 1 -- 172.21.15.70:0/3037870229 wait complete. 2024-04-15T15:15:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.914+0000 7f71158ed640 1 -- 172.21.15.70:0/230262695 reap_dead start 2024-04-15T15:15:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 shutdown_connections 2024-04-15T15:15:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 >> 172.21.15.70:0/230262695 conn(0x7f71100b5680 msgr2=0x7f7110162480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 shutdown_connections 2024-04-15T15:15:28.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.915+0000 7f7117350640 1 -- 172.21.15.70:0/230262695 wait complete. 2024-04-15T15:15:28.973 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:28.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:15:28.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.920+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 --> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4628002dd0 con 0x7f4624076140 2024-04-15T15:15:28.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.920+0000 7f4632ffd640 1 -- 172.21.15.70:0/1064522175 <== osd.3 v2:172.21.15.70:6824/1402304147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4628002dd0 con 0x7f4624076140 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.921+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7f4624076140 msgr2=0x7f4624078580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.921+0000 7f464359e640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7f4624076140 0x7f4624078580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f462406c9c0 msgr2=0x7f462406ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f462406c9c0 0x7f462406ee80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f463c048a30 tx=0x7f46380016f0 comp rx=0 tx=0).stop 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 msgr2=0x7f463c0dbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 --2- 172.21.15.70:0/1064522175 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f463c03a5c0 0x7f463c0dbc70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f463400c570 tx=0x7f463400ca30 comp rx=0 tx=0).stop 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f4642d9d640 1 -- 172.21.15.70:0/1064522175 reap_dead start 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 shutdown_connections 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 >> 172.21.15.70:0/1064522175 conn(0x7f463c009fa0 msgr2=0x7f463c0048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 shutdown_connections 2024-04-15T15:15:28.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:28.922+0000 7f464359e640 1 -- 172.21.15.70:0/1064522175 wait complete. 2024-04-15T15:15:28.976 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:28.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:15:28.977 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-15T15:15:28.977 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:15:28.978 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:28.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:15:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:28.979 INFO:teuthology.orchestra.run.smithi070.stdout:60129542154 2024-04-15T15:15:28.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:15:28.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:15:28.981 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:28.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:15:28.982 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:28.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:15:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.186+0000 7f6fcbb32640 1 Processor -- start 2024-04-15T15:15:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.186+0000 7f6fcbb32640 1 -- start start 2024-04-15T15:15:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fcbb32640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 0x7f6fc416e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fcbb32640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 0x7f6fc40b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fcbb32640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fc416d410 con 0x7f6fc416ba50 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fcbb32640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6fc416d6f0 con 0x7f6fc416e9f0 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fcbb32640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc416d9d0 con 0x7f6fc416c3f0 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fc90cd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6fc416ba50 0x7f6fc41695b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55932/0 (socket says 172.21.15.70:55932) 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fc90cd640 1 -- 172.21.15.70:0/2499944828 learned_addr learned my addr 172.21.15.70:0/2499944828 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/2499944828 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 0x7f6fc416e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.189+0000 7f6fca0cf640 1 --2- 172.21.15.70:0/2499944828 >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 0x7f6fc40b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.188+0000 7f6fc88cc640 1 -- 172.21.15.70:0/2499944828 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203815414 0 0) 0x7f6fc416d410 con 0x7f6fc416ba50 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7f6fc88cc640 1 -- 172.21.15.70:0/2499944828 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fb0003420 con 0x7f6fc416ba50 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.189+0000 7f4f8ca35640 1 Processor -- start 2024-04-15T15:15:29.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.189+0000 7f4f8ca35640 1 -- start start 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.190+0000 7f4f8ca35640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f8816ba50 0x7f4f881695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.190+0000 7f4f8ca35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 0x7f4f880b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f8ca35640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f8816d410 con 0x7f4f8816c3f0 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f8ca35640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f8816d6f0 con 0x7f4f8816ba50 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f8ca35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8816d9d0 con 0x7f4f88169af0 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f7ffff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f8816ba50 0x7f4f881695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f7ffff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f8816ba50 0x7f4f881695b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60842/0 (socket says 172.21.15.70:60842) 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f7ffff640 1 -- 172.21.15.70:0/1510199342 learned_addr learned my addr 172.21.15.70:0/1510199342 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.189+0000 7f4a096f7640 1 Processor -- start 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.189+0000 7f4a096f7640 1 -- start start 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a096f7640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4a040cf140 0x7f4a040cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a03fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4a040cf140 0x7f4a040cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a03fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4a040cf140 0x7f4a040cd0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60852/0 (socket says 172.21.15.70:60852) 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a03fff640 1 -- 172.21.15.70:0/2800013168 learned_addr learned my addr 172.21.15.70:0/2800013168 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0416b720 con 0x7f4a0406ef50 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a0416efe0 con 0x7f4a040b4240 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4a0416fda0 con 0x7f4a040cf140 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 411671333 0 0) 0x7f4a0416efe0 con 0x7f4a040b4240 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49ec003500 con 0x7f4a040b4240 2024-04-15T15:15:29.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f86d9d640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 0x7f4f880b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7f4f8659c640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f4f8816c3f0 0x7f4f8816e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55944/0 (socket says 172.21.15.70:55944) 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.190+0000 7fb8e78d9640 1 Processor -- start 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.190+0000 7fb8e78d9640 1 -- start start 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e78d9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8e00ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e78d9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e00ccfc0 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e78d9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8e00cd2a0 con 0x7fb8e00cc890 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522243121 0 0) 0x7f4a0416b720 con 0x7f4a0406ef50 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a0416efe0 con 0x7f4a0406ef50 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e78d9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e016eef0 con 0x7fb8e00af310 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e5e76640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb8e00cf1f0 0x7fb8e008b050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44756/0 (socket says 172.21.15.70:44756) 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e5e76640 1 -- 172.21.15.70:0/2054606029 learned_addr learned my addr 172.21.15.70:0/2054606029 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:29.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.191+0000 7fb8e5675640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8e00ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7ff4827ab640 1 Processor -- start 2024-04-15T15:15:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7ff4827ab640 1 -- start start 2024-04-15T15:15:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4827ab640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff474100c50 0x7ff47403a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4827ab640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff4740065f0 0x7ff474053030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4827ab640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff47403aa60 con 0x7ff474101030 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4827ab640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff474007020 con 0x7ff4740065f0 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4827ab640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff474007300 con 0x7ff474100c50 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff481faa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff4740065f0 0x7ff474053030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff4817a9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff474100c50 0x7ff47403a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff480fa8640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff474101030 0x7ff474005ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:55992/0 (socket says 172.21.15.70:55992) 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7ff480fa8640 1 -- 172.21.15.70:0/1589424226 learned_addr learned my addr 172.21.15.70:0/1589424226 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7ff473fff640 1 -- 172.21.15.70:0/1589424226 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 758930313 0 0) 0x7ff47403aa60 con 0x7ff474101030 2024-04-15T15:15:29.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7fbfddf10640 1 Processor -- start 2024-04-15T15:15:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7fbfddf10640 1 -- start start 2024-04-15T15:15:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfddf10640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd80cf140 0x7fbfd80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfddf10640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbfd80b4240 0x7fbfd80b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfddf10640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfd816b720 con 0x7fbfd806ef50 2024-04-15T15:15:29.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfddf10640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbfd816fda0 con 0x7fbfd80b4240 2024-04-15T15:15:29.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7ff473fff640 1 -- 172.21.15.70:0/1589424226 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff468003520 con 0x7ff474101030 2024-04-15T15:15:29.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfddf10640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd816f910 con 0x7fbfd80cf140 2024-04-15T15:15:29.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfdcf0e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd80cf140 0x7fbfd80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfd7fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fbfd806ef50 0x7fbfd80cf520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:56008/0 (socket says 172.21.15.70:56008) 2024-04-15T15:15:29.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7fbfd7fff640 1 -- 172.21.15.70:0/613996900 learned_addr learned my addr 172.21.15.70:0/613996900 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:29.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4004757158 0 0) 0x7fb8e00ccfc0 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8cc003500 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3587401103 0 0) 0x7fb8e00cd2a0 con 0x7fb8e00cc890 2024-04-15T15:15:29.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.241+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8e00ccfc0 con 0x7fb8e00cc890 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7fbfdd70f640 1 --2- 172.21.15.70:0/613996900 >> v2:172.21.15.96:3300/0 conn(0x7fbfd80b4240 0x7fbfd80b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7f760dcb1640 1 Processor -- start 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7f760dcb1640 1 -- start start 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f760dcb1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 0x7f760816e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f760dcb1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 0x7f76081695b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f760dcb1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f760816d410 con 0x7f760816e9f0 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f760dcb1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f760816d6f0 con 0x7f760816c3f0 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f760dcb1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f760816d9d0 con 0x7f760816ba50 2024-04-15T15:15:29.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f76077fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 0x7f760816e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f76077fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 0x7f760816e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36644/0 (socket says 172.21.15.70:36644) 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f76077fe640 1 -- 172.21.15.70:0/1349672980 learned_addr learned my addr 172.21.15.70:0/1349672980 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f76067fc640 1 -- 172.21.15.70:0/1349672980 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3141800006 0 0) 0x7f760816d410 con 0x7f760816e9f0 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f7606ffd640 1 --2- 172.21.15.70:0/1349672980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 0x7f76081695b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f76067fc640 1 -- 172.21.15.70:0/1349672980 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f75f0003500 con 0x7f760816e9f0 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1529070155 0 0) 0x7fb8cc003500 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 msgr2=0x7fb8e00ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8e00ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 >> v1:172.21.15.96:6789/0 conn(0x7fb8e00cc890 legacy=0x7fb8e016f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e016f1d0 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7fbfd77fe640 1 -- 172.21.15.70:0/613996900 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2095285129 0 0) 0x7fbfd816b720 con 0x7fbfd806ef50 2024-04-15T15:15:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fbfd77fe640 1 -- 172.21.15.70:0/613996900 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfc8003520 con 0x7fbfd806ef50 2024-04-15T15:15:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4f7ffff640 1 -- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 msgr2=0x7f4f880b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7f19ed0a0640 1 Processor -- start 2024-04-15T15:15:29.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.192+0000 7f19ed0a0640 1 -- start start 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4f7ffff640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 0x7f4f880b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19ed0a0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f19e80af310 0x7f19e80ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19ed0a0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f19e80cc890 0x7f19e816f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19ed0a0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19e80ccd00 con 0x7f19e80cf1f0 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19ed0a0640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f19e80ccfc0 con 0x7f19e80cc890 2024-04-15T15:15:29.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19ed0a0640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f19e80cd2a0 con 0x7f19e80af310 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19e6d9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f19e80af310 0x7f19e80ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19e6d9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f19e80af310 0x7f19e80ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60868/0 (socket says 172.21.15.70:60868) 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19e759e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f19e80cf1f0 0x7f19e808b050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44758/0 (socket says 172.21.15.70:44758) 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19e6d9d640 1 -- 172.21.15.70:0/3509902325 learned_addr learned my addr 172.21.15.70:0/3509902325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4f7ffff640 1 -- 172.21.15.70:0/1510199342 >> v1:172.21.15.96:6789/0 conn(0x7f4f8816c3f0 legacy=0x7f4f8816e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4f7ffff640 1 -- 172.21.15.70:0/1510199342 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f8816dcb0 con 0x7f4f8816ba50 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f6fc98ce640 1 -- 172.21.15.70:0/2499944828 >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 msgr2=0x7f6fc40b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/2499944828 >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 0x7f6fc40b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f6fc98ce640 1 -- 172.21.15.70:0/2499944828 >> v1:172.21.15.96:6789/0 conn(0x7f6fc416ba50 legacy=0x7f6fc41695b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f6fc98ce640 1 -- 172.21.15.70:0/2499944828 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc416dcb0 con 0x7f6fc416c3f0 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fb8d80019f0 con 0x7fb8e00cf1f0 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 >> v1:172.21.15.70:6789/0 conn(0x7fb8e00cf1f0 legacy=0x7fb8e008b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc003cd0 0x7fb8cc0040b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8cc0045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc00d410 0x7fb8cc00f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e016eef0 con 0x7fb8cc003cd0 2024-04-15T15:15:29.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4f86d9d640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 0x7f4f880b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc006f80 con 0x7fb8cc00d410 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8cc007180 con 0x7fb8e00af310 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3446582245 0 0) 0x7f49ec003500 con 0x7f4a040b4240 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 >> v2:172.21.15.96:3300/0 conn(0x7f4a040cf140 msgr2=0x7f4a040cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 --2- 172.21.15.70:0/2800013168 >> v2:172.21.15.96:3300/0 conn(0x7f4a040cf140 0x7f4a040cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 >> v1:172.21.15.70:6789/0 conn(0x7f4a0406ef50 legacy=0x7f4a040cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0416f2c0 con 0x7f4a040b4240 2024-04-15T15:15:29.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7ff481faa640 1 -- 172.21.15.70:0/1589424226 >> v2:172.21.15.96:3300/0 conn(0x7ff474100c50 msgr2=0x7ff47403a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7ff481faa640 1 --2- 172.21.15.70:0/1589424226 >> v2:172.21.15.96:3300/0 conn(0x7ff474100c50 0x7ff47403a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7ff481faa640 1 -- 172.21.15.70:0/1589424226 >> v1:172.21.15.96:6789/0 conn(0x7ff474101030 legacy=0x7ff474005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7ff481faa640 1 -- 172.21.15.70:0/1589424226 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4740075e0 con 0x7ff4740065f0 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f49fc0019f0 con 0x7f4a040b4240 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 >> v1:172.21.15.96:6789/0 conn(0x7f4a040b4240 legacy=0x7f4a040b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49ec003cd0 0x7f49ec0040d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec004680 0x7f49ec00c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 0x7f49ec00e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00f220 con 0x7f49ec004680 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec00f3d0 con 0x7f49ec00c560 2024-04-15T15:15:29.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f49ec00f5d0 con 0x7f49ec003cd0 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f7606ffd640 1 -- 172.21.15.70:0/1349672980 >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 msgr2=0x7f760816e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f7606ffd640 1 --2- 172.21.15.70:0/1349672980 >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 0x7f760816e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f7606ffd640 1 -- 172.21.15.70:0/1349672980 >> v1:172.21.15.96:6789/0 conn(0x7f760816e9f0 legacy=0x7f76080b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7f7606ffd640 1 -- 172.21.15.70:0/1349672980 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f760816dcb0 con 0x7f760816ba50 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc00d410 0x7fb8cc00f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e4e74640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8cc0045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5675640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc003cd0 0x7fb8cc0040b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f6fca0cf640 1 --2- 172.21.15.70:0/2499944828 >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 0x7f6fc40b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/2499944828 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 0x7f6fc416e4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc008d70 tx=0x7f6fbc02bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51aed9d096020227 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.193+0000 7f19e659c640 1 --2- 172.21.15.70:0/3509902325 >> v2:172.21.15.70:3300/0 conn(0x7f19e80cc890 0x7f19e816f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.194+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812586808 0 0) 0x7f19e80ccd00 con 0x7f19e80cf1f0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f19d0003520 con 0x7f19e80cf1f0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1819591475 0 0) 0x7f19d0003520 con 0x7f19e80cf1f0 2024-04-15T15:15:29.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 >> v2:172.21.15.96:3300/0 conn(0x7f19e80af310 msgr2=0x7f19e80ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 --2- 172.21.15.70:0/3509902325 >> v2:172.21.15.96:3300/0 conn(0x7f19e80af310 0x7f19e80ce9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 >> v2:172.21.15.70:3300/0 conn(0x7f19e80cc890 msgr2=0x7f19e816f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 --2- 172.21.15.70:0/3509902325 >> v2:172.21.15.70:3300/0 conn(0x7f19e80cc890 0x7f19e816f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 0x7f49ec00e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e816eef0 con 0x7f19e80cf1f0 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f19e00019f0 con 0x7f19e80cf1f0 2024-04-15T15:15:29.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 >> v1:172.21.15.70:6789/0 conn(0x7f19e80cf1f0 legacy=0x7f19e808b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19d00099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fbfdd70f640 1 -- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd80cf140 msgr2=0x7fbfd80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fbfdd70f640 1 --2- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd80cf140 0x7fbfd80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.242+0000 7fbfdd70f640 1 -- 172.21.15.70:0/613996900 >> v1:172.21.15.96:6789/0 conn(0x7fbfd806ef50 legacy=0x7fbfd80cf520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fbfdd70f640 1 -- 172.21.15.70:0/613996900 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd816efe0 con 0x7fbfd80b4240 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d0004800 0x7f19d000d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80ccfc0 con 0x7f19d00039b0 2024-04-15T15:15:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80cd2a0 con 0x7f19d0004800 2024-04-15T15:15:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f19d0004cb0 con 0x7f19d0005090 2024-04-15T15:15:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a03fff640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49ec003cd0 0x7f49ec0040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f6fc88cc640 1 -- 172.21.15.70:0/2499944828 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc039070 con 0x7f6fc416c3f0 2024-04-15T15:15:29.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 -- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 msgr2=0x7fb8cc0045f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e00af310 0x7fb8cc0045f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 -- 172.21.15.70:0/2054606029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc003cd0 msgr2=0x7fb8cc0040b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f6fc88cc640 1 -- 172.21.15.70:0/2499944828 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fbc02c8f0 con 0x7f6fc416c3f0 2024-04-15T15:15:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc003cd0 0x7fb8cc0040b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.243+0000 7fb8e5e76640 1 -- 172.21.15.70:0/2054606029 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc003500 con 0x7fb8cc00d410 2024-04-15T15:15:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f7606ffd640 1 --2- 172.21.15.70:0/1349672980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 0x7f76081695b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0061f0 tx=0x7f75fc02c0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94c40c98d320535e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a037fe640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec004680 0x7f49ec00c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19d00099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e6d9d640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7ff481faa640 1 --2- 172.21.15.70:0/1589424226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4740065f0 0x7ff474053030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff47c051700 tx=0x7ff47c097520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295ce65e7ebde24 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e759e640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d0004800 0x7f19d000d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4f7ffff640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8816ba50 0x7f4f881695b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007860 tx=0x7f4f7002cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42940ebd02584486 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f76067fc640 1 -- 172.21.15.70:0/1349672980 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75fc039070 con 0x7f760816ba50 2024-04-15T15:15:29.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f76067fc640 1 -- 172.21.15.70:0/1349672980 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75fc02cc20 con 0x7f760816ba50 2024-04-15T15:15:29.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f76067fc640 1 -- 172.21.15.70:0/1349672980 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76000019f0 con 0x7f760816ba50 2024-04-15T15:15:29.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f6fc88cc640 1 -- 172.21.15.70:0/2499944828 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fc0001670 con 0x7f6fc416c3f0 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 -- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d0004800 msgr2=0x7f19d000d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19d0004800 0x7f19d000d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 -- 172.21.15.70:0/3509902325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d00039b0 msgr2=0x7f19d0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d00039b0 0x7f19d0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e659c640 1 -- 172.21.15.70:0/3509902325 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e816eef0 con 0x7f19d0005090 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7ff473fff640 1 -- 172.21.15.70:0/1589424226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c072040 con 0x7ff4740065f0 2024-04-15T15:15:29.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7ff473fff640 1 -- 172.21.15.70:0/1589424226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47c09c7d0 con 0x7ff4740065f0 2024-04-15T15:15:29.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc00d410 0x7fb8cc00f800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800a7c0 tx=0x7fb8cc004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc24618f2073c239 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f19e759e640 1 -- 172.21.15.70:0/3509902325 reap_dead start 2024-04-15T15:15:29.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7ff473fff640 1 -- 172.21.15.70:0/1589424226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c09cc40 con 0x7ff4740065f0 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a08ef6640 1 -- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49ec003cd0 msgr2=0x7f49ec0040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49ec003cd0 0x7f49ec0040d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a08ef6640 1 -- 172.21.15.70:0/2800013168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec004680 msgr2=0x7f49ec00c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec004680 0x7f49ec00c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7f4a08ef6640 1 -- 172.21.15.70:0/2800013168 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0416f2c0 con 0x7f49ec00c560 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7fbfdd70f640 1 --2- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd80b46d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0006130 tx=0x7fbfd002e440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce579fd57c21d85c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.244+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d800a410 con 0x7fb8cc00d410 2024-04-15T15:15:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8d8034650 con 0x7fb8cc00d410 2024-04-15T15:15:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fb8d7fff640 1 -- 172.21.15.70:0/2054606029 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8d8034a40 con 0x7fb8cc00d410 2024-04-15T15:15:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 -- 172.21.15.70:0/2499944828 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 msgr2=0x7f6fc416e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/2499944828 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 0x7f6fc416e4b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc008d70 tx=0x7f6fbc02bdc0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fbfd77fe640 1 -- 172.21.15.70:0/613996900 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd003a070 con 0x7fbfd80b4240 2024-04-15T15:15:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4f85d9b640 1 -- 172.21.15.70:0/1510199342 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7002d980 con 0x7f4f8816ba50 2024-04-15T15:15:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4f85d9b640 1 -- 172.21.15.70:0/1510199342 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f7000ae50 con 0x7f4f8816ba50 2024-04-15T15:15:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fbfd77fe640 1 -- 172.21.15.70:0/613996900 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd0003b90 con 0x7fbfd80b4240 2024-04-15T15:15:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fbfd77fe640 1 -- 172.21.15.70:0/613996900 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0004030 con 0x7fbfd80b4240 2024-04-15T15:15:29.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4f85d9b640 1 -- 172.21.15.70:0/1510199342 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7002dd40 con 0x7f4f8816ba50 2024-04-15T15:15:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4a08ef6640 1 -- 172.21.15.70:0/2800013168 reap_dead start 2024-04-15T15:15:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 0x7f49ec00e970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c4e0 tx=0x7f49ec004d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7e62c58513d9971 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc039070 con 0x7f49ec00c560 2024-04-15T15:15:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49fc02cd50 con 0x7f49ec00c560 2024-04-15T15:15:29.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f4a02ffd640 1 -- 172.21.15.70:0/2800013168 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49fc00c0e0 con 0x7f49ec00c560 2024-04-15T15:15:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 -- 172.21.15.70:0/2499944828 shutdown_connections 2024-04-15T15:15:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/2499944828 >> v2:172.21.15.96:3300/0 conn(0x7f6fc416e9f0 0x7f6fc40b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/2499944828 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416c3f0 0x7f6fc416e4b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f760dcb1640 1 -- 172.21.15.70:0/1349672980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 msgr2=0x7f76081695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f760dcb1640 1 --2- 172.21.15.70:0/1349672980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 0x7f76081695b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f75fc0061f0 tx=0x7f75fc02c0f0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fb8e78d9640 1 -- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc00d410 msgr2=0x7fb8cc00f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/2054606029 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8cc00d410 0x7fb8cc00f800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb8d800a7c0 tx=0x7fb8cc004d50 comp rx=0 tx=0).stop 2024-04-15T15:15:29.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7fb8e5e76640 1 -- 172.21.15.70:0/2054606029 reap_dead start 2024-04-15T15:15:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 -- 172.21.15.70:0/2499944828 >> 172.21.15.70:0/2499944828 conn(0x7f6fc40b5680 msgr2=0x7f6fc40b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 -- 172.21.15.70:0/2499944828 shutdown_connections 2024-04-15T15:15:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.245+0000 7f6fcbb32640 1 -- 172.21.15.70:0/2499944828 wait complete. 2024-04-15T15:15:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 -- 172.21.15.70:0/1589424226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4740065f0 msgr2=0x7ff474053030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 --2- 172.21.15.70:0/1589424226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4740065f0 0x7ff474053030 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff47c051700 tx=0x7ff47c097520 comp rx=0 tx=0).stop 2024-04-15T15:15:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 -- 172.21.15.70:0/1349672980 shutdown_connections 2024-04-15T15:15:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 --2- 172.21.15.70:0/1349672980 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816ba50 0x7f76081695b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 --2- 172.21.15.70:0/1349672980 >> v2:172.21.15.70:3300/0 conn(0x7f760816c3f0 0x7f760816e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 -- 172.21.15.70:0/1349672980 >> 172.21.15.70:0/1349672980 conn(0x7f76080b5680 msgr2=0x7f76080b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 -- 172.21.15.70:0/1589424226 shutdown_connections 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 --2- 172.21.15.70:0/1589424226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff4740065f0 0x7ff474053030 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 --2- 172.21.15.70:0/1589424226 >> v2:172.21.15.96:3300/0 conn(0x7ff474100c50 0x7ff47403a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 -- 172.21.15.70:0/1589424226 >> 172.21.15.70:0/1589424226 conn(0x7ff474009fb0 msgr2=0x7ff47400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 -- 172.21.15.70:0/1349672980 shutdown_connections 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 -- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 msgr2=0x7fbfd80b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f760dcb1640 1 -- 172.21.15.70:0/1349672980 wait complete. 2024-04-15T15:15:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 -- 172.21.15.70:0/1589424226 shutdown_connections 2024-04-15T15:15:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7ff4827ab640 1 -- 172.21.15.70:0/1589424226 wait complete. 2024-04-15T15:15:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 --2- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd80b46d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbfd0006130 tx=0x7fbfd002e440 comp rx=0 tx=0).stop 2024-04-15T15:15:29.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 -- 172.21.15.70:0/613996900 shutdown_connections 2024-04-15T15:15:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 msgr2=0x7f49ec00e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 0x7f49ec00e970 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00c4e0 tx=0x7f49ec004d50 comp rx=0 tx=0).stop 2024-04-15T15:15:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 --2- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd80b46d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 --2- 172.21.15.70:0/613996900 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd80cf140 0x7fbfd80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fbfddf10640 1 -- 172.21.15.70:0/613996900 >> 172.21.15.70:0/613996900 conn(0x7fbfd80b1c50 msgr2=0x7fbfd80b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 shutdown_connections 2024-04-15T15:15:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 --2- 172.21.15.70:0/2800013168 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49ec00c560 0x7f49ec00e970 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 >> 172.21.15.70:0/2800013168 conn(0x7f4a040b1c50 msgr2=0x7f4a040b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fb8e78d9640 1 -- 172.21.15.70:0/2054606029 shutdown_connections 2024-04-15T15:15:29.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7fb8e78d9640 1 -- 172.21.15.70:0/2054606029 >> 172.21.15.70:0/2054606029 conn(0x7fb8e00b1c50 msgr2=0x7fb8e00b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4f8ca35640 1 -- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8816ba50 msgr2=0x7f4f881695b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 shutdown_connections 2024-04-15T15:15:29.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f19e659c640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19d00099b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ac0 tx=0x7f19d0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11cd2e3c23f86541 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7fbfddf10640 1 -- 172.21.15.70:0/613996900 shutdown_connections 2024-04-15T15:15:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7fbfddf10640 1 -- 172.21.15.70:0/613996900 wait complete. 2024-04-15T15:15:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7fb8e78d9640 1 -- 172.21.15.70:0/2054606029 shutdown_connections 2024-04-15T15:15:29.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7fb8e78d9640 1 -- 172.21.15.70:0/2054606029 wait complete. 2024-04-15T15:15:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.246+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8816ba50 0x7f4f881695b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4f70007860 tx=0x7f4f7002cbd0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f4f8ca35640 1 -- 172.21.15.70:0/1510199342 shutdown_connections 2024-04-15T15:15:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc039070 con 0x7f19d0005090 2024-04-15T15:15:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19dc02c6b0 con 0x7f19d0005090 2024-04-15T15:15:29.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f19e5d9b640 1 -- 172.21.15.70:0/3509902325 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19dc009690 con 0x7f19d0005090 2024-04-15T15:15:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f88169af0 0x7f4f880b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/1510199342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f8816ba50 0x7f4f881695b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f4f8ca35640 1 -- 172.21.15.70:0/1510199342 >> 172.21.15.70:0/1510199342 conn(0x7f4f880b5680 msgr2=0x7f4f880b5a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f6fcbb32640 1 Processor -- start 2024-04-15T15:15:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f4a096f7640 1 -- 172.21.15.70:0/2800013168 wait complete. 2024-04-15T15:15:29.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.247+0000 7f6fcbb32640 1 -- start start 2024-04-15T15:15:29.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f4f8ca35640 1 -- 172.21.15.70:0/1510199342 shutdown_connections 2024-04-15T15:15:29.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f4f8ca35640 1 -- 172.21.15.70:0/1510199342 wait complete. 2024-04-15T15:15:29.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 -- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 msgr2=0x7f19d00099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19d00099b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f19dc009ac0 tx=0x7f19d0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 -- 172.21.15.70:0/3509902325 shutdown_connections 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 --2- 172.21.15.70:0/3509902325 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19d00099b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 -- 172.21.15.70:0/3509902325 >> 172.21.15.70:0/3509902325 conn(0x7f19e80b1c50 msgr2=0x7f19e80b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 -- 172.21.15.70:0/3509902325 shutdown_connections 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f19ed0a0640 1 -- 172.21.15.70:0/3509902325 wait complete. 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7f4a096f7640 1 Processor -- start 2024-04-15T15:15:29.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7f760dcb1640 1 Processor -- start 2024-04-15T15:15:29.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7f4a096f7640 1 -- start start 2024-04-15T15:15:29.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7fbfddf10640 1 Processor -- start 2024-04-15T15:15:29.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7f6fcbb32640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7fb8e78d9640 1 Processor -- start 2024-04-15T15:15:29.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7fb8e78d9640 1 -- start start 2024-04-15T15:15:29.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7fbfddf10640 1 -- start start 2024-04-15T15:15:29.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7f6fcbb32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 0x7f6fc409d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7ff4827ab640 1 Processor -- start 2024-04-15T15:15:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.248+0000 7ff4827ab640 1 -- start start 2024-04-15T15:15:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7ff4827ab640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 0x7ff474046130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7ff4827ab640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.296 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7ff4827ab640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 0x7ff4741a28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fc98ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f760dcb1640 1 -- start start 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fc98ce640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60906/0 (socket says 172.21.15.70:60906) 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fc98ce640 1 -- 172.21.15.70:0/3644526700 learned_addr learned my addr 172.21.15.70:0/3644526700 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416e9f0 0x7f6fc409d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4169e40 con 0x7f6fc416c3f0 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f4f8ca35640 1 Processor -- start 2024-04-15T15:15:29.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fc90cd640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 0x7f6fc409d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7fb8e78d9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f4f8ca35640 1 -- start start 2024-04-15T15:15:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4817a9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 0x7ff474046130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4817a9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 0x7ff474046130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42954/0 (socket says 172.21.15.70:42954) 2024-04-15T15:15:29.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.249+0000 7ff480fa8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6fc4169cc0 con 0x7f6fc416ba50 2024-04-15T15:15:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6fc40b36a0 con 0x7f6fc416e9f0 2024-04-15T15:15:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff480fa8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36696/0 (socket says 172.21.15.70:36696) 2024-04-15T15:15:29.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4817a9640 1 -- 172.21.15.70:0/1925134978 learned_addr learned my addr 172.21.15.70:0/1925134978 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f4a096f7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff481faa640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 0x7ff4741a28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7f4a096f7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 0x7f4a04208d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4827ab640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff474053570 con 0x7ff474100c50 2024-04-15T15:15:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4827ab640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff474006c90 con 0x7ff474101030 2024-04-15T15:15:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7ff4827ab640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff4740536f0 con 0x7ff4740065f0 2024-04-15T15:15:29.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7fbfddf10640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 0x7fbfd808b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.250+0000 7fb8e5675640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a03fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36710/0 (socket says 172.21.15.70:36710) 2024-04-15T15:15:29.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a096f7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 0x7f4a0409ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f19ed0a0640 1 Processor -- start 2024-04-15T15:15:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfddf10640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd809b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a03fff640 1 -- 172.21.15.70:0/3843948987 learned_addr learned my addr 172.21.15.70:0/3843948987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 0x7f7608094780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816c3f0 0x7f76080963d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfdcf0e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 0x7fbfd808b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f76080b39f0 con 0x7f760816ba50 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f76080b3870 con 0x7f760816e9f0 2024-04-15T15:15:29.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f760dcb1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7608169af0 con 0x7f760816c3f0 2024-04-15T15:15:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0406f3b0 con 0x7f49ec00c560 2024-04-15T15:15:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0416f910 con 0x7f4a042092d0 2024-04-15T15:15:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4a040cd0a0 con 0x7f4a04164c80 2024-04-15T15:15:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfdcf0e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 0x7fbfd808b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42976/0 (socket says 172.21.15.70:42976) 2024-04-15T15:15:29.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f19ed0a0640 1 -- start start 2024-04-15T15:15:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfdcf0e640 1 -- 172.21.15.70:0/1874274535 learned_addr learned my addr 172.21.15.70:0/1874274535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f7607fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfddf10640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 0x7fbfd8072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e78d9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e0073530 0x7fb8e0212590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e78d9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 0x7fb8e00ae0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e78d9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00af870 con 0x7fb8cc00d410 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e78d9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e00af6f0 con 0x7fb8e0073530 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e78d9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e008b760 con 0x7fb8e0212ad0 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36706/0 (socket says 172.21.15.70:36706) 2024-04-15T15:15:29.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 -- 172.21.15.70:0/4183144125 learned_addr learned my addr 172.21.15.70:0/4183144125 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f7607fff640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60940/0 (socket says 172.21.15.70:60940) 2024-04-15T15:15:29.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f7607fff640 1 -- 172.21.15.70:0/645563082 learned_addr learned my addr 172.21.15.70:0/645563082 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4f8ca35640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f6fca0cf640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416e9f0 0x7f6fc409d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfd7fff640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd809b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 0x7f4a0409ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4a037fe640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 0x7f4a04208d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4f8ca35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 0x7f4f880cf370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f76077fe640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 0x7f7608094780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4f8ca35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fbfdd70f640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 0x7fbfd8072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fbfddf10640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd806f3b0 con 0x7fbfd80cf140 2024-04-15T15:15:29.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f7ffff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 0x7f4f880cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f7ffff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 0x7f4f880cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42998/0 (socket says 172.21.15.70:42998) 2024-04-15T15:15:29.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f8659c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:60968/0 (socket says 172.21.15.70:60968) 2024-04-15T15:15:29.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7606ffd640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816c3f0 0x7f76080963d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 0x7fb8e00ae0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e4e74640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e0073530 0x7fb8e0212590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 -- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 msgr2=0x7fb8e00ae0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fbfddf10640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd816f5a0 con 0x7fbfd80b4240 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 0x7fb8e00ae0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 -- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e0073530 msgr2=0x7fb8e0212590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e0073530 0x7fb8e0212590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.251+0000 7fb8e5675640 1 -- 172.21.15.70:0/4183144125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e0072940 con 0x7fb8cc00d410 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8e5e76640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 0x7fb8e00ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8e5675640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007a50 tx=0x7fb8dc007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 -- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816c3f0 msgr2=0x7f76080963d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816c3f0 0x7f76080963d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 -- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 msgr2=0x7f7608094780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 0x7f7608094780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f86d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f86d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36746/0 (socket says 172.21.15.70:36746) 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 -- 172.21.15.70:0/645563082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7608096db0 con 0x7f760816e9f0 2024-04-15T15:15:29.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f76077fe640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 0x7f7608094780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc98ce640 1 -- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416e9f0 msgr2=0x7f6fc409d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416e9f0 0x7f6fc409d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc98ce640 1 -- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 msgr2=0x7f6fc409d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 0x7f6fc409d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc98ce640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fc40816c0 con 0x7f6fc416ba50 2024-04-15T15:15:29.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f6fc90cd640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 0x7f6fc409d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f8ca35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8816ed40 con 0x7f4f8816c3f0 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 -- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 msgr2=0x7ff474046130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 0x7ff474046130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 -- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 msgr2=0x7ff474046670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f8ca35640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8816ebc0 con 0x7f4f88169af0 2024-04-15T15:15:29.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f8ca35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f880b36a0 con 0x7f4f8816ba50 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc01a020 con 0x7fb8cc00d410 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8dc010d10 con 0x7fb8cc00d410 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8e00720a0 con 0x7fb8cc00d410 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 -- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 msgr2=0x7f4a04208d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 0x7f4a04208d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 -- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 msgr2=0x7f4a0409ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 0x7f4a0409ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a04163ec0 con 0x7f49ec00c560 2024-04-15T15:15:29.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a08ef6640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 0x7f4a0409ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a037fe640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 0x7f4a04208d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4f7ffff640 1 -- 172.21.15.70:0/2365238331 learned_addr learned my addr 172.21.15.70:0/2365238331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4740438e0 con 0x7ff474101030 2024-04-15T15:15:29.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fbfddf10640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfd80b4c10 con 0x7fbfd806ef50 2024-04-15T15:15:29.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff480fa8640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:29.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f19ed0a0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f19ed0a0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f4a03fff640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011a10 tx=0x7f49f4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7ff481faa640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 0x7ff4741a28a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0654e0 tx=0x7ff47c069760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f7607fff640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f760000ca10 tx=0x7f760000ced0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f19e6d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8dc015680 con 0x7fb8cc00d410 2024-04-15T15:15:29.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.252+0000 7f19e6d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43022/0 (socket says 172.21.15.70:43022) 2024-04-15T15:15:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19ed0a0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19e8167e90 0x7f19e81f53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8e0072660 con 0x7fb8cc00d410 2024-04-15T15:15:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36758/0 (socket says 172.21.15.70:36758) 2024-04-15T15:15:29.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 -- 172.21.15.70:0/187813266 learned_addr learned my addr 172.21.15.70:0/187813266 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:29.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f4019070 con 0x7f49ec00c560 2024-04-15T15:15:29.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f6fc98ce640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc000f80 tx=0x7f6fbc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7600003d80 con 0x7f760816e9f0 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7600015070 con 0x7f760816e9f0 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760000a9a0 con 0x7f760816e9f0 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80af870 con 0x7f19e8204b00 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f19e80af6f0 con 0x7f19e8167e90 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f19e80ce9c0 con 0x7f19d0005090 2024-04-15T15:15:29.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e759e640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19e8167e90 0x7f19e81f53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 -- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 msgr2=0x7fbfd808b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 0x7fbfd808b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 -- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 msgr2=0x7fbfd8072a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 0x7fbfd8072a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd808b590 con 0x7fbfd80b4240 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfdd70f640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 0x7fbfd8072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7fbfd7fff640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd809b470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c430 tx=0x7fbfc400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4f8659c640 1 -- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 msgr2=0x7f4f880cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4f8659c640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 0x7f4f880cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4f8659c640 1 -- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 msgr2=0x7f4f880cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4f8659c640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4f8659c640 1 -- 172.21.15.70:0/2365238331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f88097c20 con 0x7f4f88169af0 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4f8659c640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800b360 tx=0x7f4f7800b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c09c5d0 con 0x7ff474101030 2024-04-15T15:15:29.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47c063260 con 0x7ff474101030 2024-04-15T15:15:29.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a041641a0 con 0x7f49ec00c560 2024-04-15T15:15:29.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a04164760 con 0x7f49ec00c560 2024-04-15T15:15:29.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff47c09fcc0 con 0x7ff474101030 2024-04-15T15:15:29.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49f40092d0 con 0x7f49ec00c560 2024-04-15T15:15:29.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7608098840 con 0x7f760816e9f0 2024-04-15T15:15:29.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7608094cc0 con 0x7f760816e9f0 2024-04-15T15:15:29.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8e004fd50 con 0x7fb8cc00d410 2024-04-15T15:15:29.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 -- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19e8167e90 msgr2=0x7f19e81f53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19e8167e90 0x7f19e81f53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 -- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 msgr2=0x7f19e8167670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19e8205310 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.253+0000 7f19e6d9d640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19d80117d0 tx=0x7f19d8011c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d8004440 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19d800f790 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19d800fb80 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19e82055f0 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f19e80ad180 con 0x7f19d0005090 2024-04-15T15:15:29.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f19e659c640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0404fd50 con 0x7f49ec00c560 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49f4023060 con 0x7f49ec00c560 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4017070 con 0x7fbfd80b4240 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc4005ce0 con 0x7fbfd80b4240 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc40072d0 con 0x7fbfd80b4240 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7fbfddf10640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd815bfe0 con 0x7fbfd80b4240 2024-04-15T15:15:29.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7fbfddf10640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd815c5a0 con 0x7fbfd80b4240 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7ff4827ab640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff474043bc0 con 0x7ff474101030 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7ff4827ab640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff474044180 con 0x7ff474101030 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc042020 con 0x7f6fc416ba50 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fbc004240 con 0x7f6fc416ba50 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc039040 con 0x7f6fc416ba50 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fc4081960 con 0x7f6fc416ba50 2024-04-15T15:15:29.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.256+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fc4081f20 con 0x7f6fc416ba50 2024-04-15T15:15:29.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.256+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4740fb840 con 0x7ff474101030 2024-04-15T15:15:29.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4f86d9d640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:29.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f78012b80 con 0x7f4f88169af0 2024-04-15T15:15:29.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f88097f00 con 0x7f4f88169af0 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.254+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f880984c0 con 0x7f4f88169af0 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f78012d20 con 0x7f4f88169af0 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.255+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7801b640 con 0x7f4f88169af0 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f49f401c070 con 0x7f49ec00c560 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8dc01d020 con 0x7fb8cc00d410 2024-04-15T15:15:29.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7fb8d5ffb640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 0x7fb8c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7f4a00ff9640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 0x7f49f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f49f408c2c0 con 0x7f49ec00c560 2024-04-15T15:15:29.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb8dc08c520 con 0x7fb8cc00d410 2024-04-15T15:15:29.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f760000ab40 con 0x7f760816e9f0 2024-04-15T15:15:29.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19e804fd50 con 0x7f19d0005090 2024-04-15T15:15:29.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f4a037fe640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 0x7f49f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.257+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfd804fd50 con 0x7fbfd80b4240 2024-04-15T15:15:29.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f19d800fd20 con 0x7f19d0005090 2024-04-15T15:15:29.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f4f7cff9640 1 -- 172.21.15.70:0/2365238331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f8816b080 con 0x7f4f88169af0 2024-04-15T15:15:29.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f7801b7e0 con 0x7f4f88169af0 2024-04-15T15:15:29.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f75effff640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 0x7f75f406eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f760008ffa0 con 0x7f760816e9f0 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.258+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76080689e0 con 0x7f760816e9f0 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f4a037fe640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 0x7f49f006edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4a040b0b30 tx=0x7f49f8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f19cf7fe640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 0x7f19d406ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f19d808c3b0 con 0x7f19d0005090 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f19e659c640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 0x7f19d406ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f4f7effd640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 0x7f4f68077610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6fbc045020 con 0x7f6fc416ba50 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fc40689e0 con 0x7f6fc416ba50 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f6fba7fc640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 0x7f6fa806ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6fbc0ad970 con 0x7f6fc416ba50 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f6fc90cd640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 0x7f6fa806ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7f6fc90cd640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 0x7f6fa806ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40d0690 tx=0x7f6fc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfc401a070 con 0x7fbfd80b4240 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.259+0000 7f4f7ffff640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 0x7f4f68077610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4f7808b5e0 con 0x7f4f88169af0 2024-04-15T15:15:29.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7f4f7ffff640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 0x7f4f68077610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000c00 tx=0x7f4f70031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7f76077fe640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 0x7f75f406eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7fb8e4e74640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 0x7fb8c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7fbfd57fa640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 0x7fbfbc073290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbfc408b1c0 con 0x7fbfd80b4240 2024-04-15T15:15:29.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff47c0b1030 con 0x7ff474101030 2024-04-15T15:15:29.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7ff471ffb640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 0x7ff46006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:29.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.260+0000 7fbfdcf0e640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 0x7fbfbc073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7f76077fe640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 0x7f75f406eeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009d40 tx=0x7f75f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7ff4817a9640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 0x7ff46006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:29.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7fbfdcf0e640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 0x7fbfbc073290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009af0 tx=0x7fbfcc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7f19e659c640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 0x7f19d406ee00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19dc036000 tx=0x7f19dc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff47c117d70 con 0x7ff474101030 2024-04-15T15:15:29.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7fb8e4e74640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 0x7fb8c406ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00b3fc0 tx=0x7fb8d0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.261+0000 7ff4817a9640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 0x7ff46006ee80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff478006fd0 tx=0x7ff478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:29.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.263+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8dc024070 con 0x7fb8cc00d410 2024-04-15T15:15:29.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.264+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7608098840 con 0x7f760816e9f0 2024-04-15T15:15:29.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.265+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f19d8019020 con 0x7f19d0005090 2024-04-15T15:15:29.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.269+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f8816b080 con 0x7f4f88169af0 2024-04-15T15:15:29.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.270+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f49f4010040 con 0x7f49ec00c560 2024-04-15T15:15:29.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.273+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6fbc04c030 con 0x7f6fc416ba50 2024-04-15T15:15:29.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.277+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff4740fb840 con 0x7ff474101030 2024-04-15T15:15:29.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.283+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfc4058c40 con 0x7fbfd80b4240 2024-04-15T15:15:29.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.681+0000 7f4f7cff9640 1 -- 172.21.15.70:0/2365238331 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4f880689e0 con 0x7f4f88169af0 2024-04-15T15:15:29.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.682+0000 7f4f7effd640 1 -- 172.21.15.70:0/2365238331 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4f78017030 con 0x7f4f88169af0 2024-04-15T15:15:29.684 INFO:teuthology.orchestra.run.smithi070.stdout:55834574855 2024-04-15T15:15:29.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.685+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 msgr2=0x7f4f68077610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 0x7f4f68077610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4f70000c00 tx=0x7f4f70031000 comp rx=0 tx=0).stop 2024-04-15T15:15:29.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 msgr2=0x7f4f880973f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4f7800b360 tx=0x7f4f7800b820 comp rx=0 tx=0).stop 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 shutdown_connections 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f68075150 0x7f4f68077610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f8816c3f0 0x7f4f880cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f8816ba50 0x7f4f880cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.686+0000 7f4f8ca35640 1 --2- 172.21.15.70:0/2365238331 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f88169af0 0x7f4f880973f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.687+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 >> 172.21.15.70:0/2365238331 conn(0x7f4f880b5680 msgr2=0x7f4f8806e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.687+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 shutdown_connections 2024-04-15T15:15:29.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.687+0000 7f4f8ca35640 1 -- 172.21.15.70:0/2365238331 wait complete. 2024-04-15T15:15:29.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.687+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f7608072900 con 0x7f760816e9f0 2024-04-15T15:15:29.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.688+0000 7f75effff640 1 -- 172.21.15.70:0/645563082 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f760005e070 con 0x7f760816e9f0 2024-04-15T15:15:29.690 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.688+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fb8e00cdc20 con 0x7fb8cc00d410 2024-04-15T15:15:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.689+0000 7fb8d5ffb640 1 -- 172.21.15.70:0/4183144125 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8dc05a600 con 0x7fb8cc00d410 2024-04-15T15:15:29.691 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:29.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.689+0000 7f19ed0a0640 1 -- 172.21.15.70:0/187813266 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f19e80ccd00 con 0x7f19d0005090 2024-04-15T15:15:29.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.690+0000 7f19cf7fe640 1 -- 172.21.15.70:0/187813266 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f19d8010040 con 0x7f19d0005090 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f6fc40d0780 con 0x7f6fc416ba50 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 msgr2=0x7f75f406eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 0x7f75f406eeb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009d40 tx=0x7f75f8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 msgr2=0x7f760806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f760000ca10 tx=0x7f760000ced0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 shutdown_connections 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f760816e9f0 0x7f760806ddb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f760816c3f0 0x7f76080963d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f75f406c9f0 0x7f75f406eeb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 --2- 172.21.15.70:0/645563082 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f760816ba50 0x7f7608094780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 >> 172.21.15.70:0/645563082 conn(0x7f76080b5680 msgr2=0x7f760816a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 shutdown_connections 2024-04-15T15:15:29.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f760dcb1640 1 -- 172.21.15.70:0/645563082 wait complete. 2024-04-15T15:15:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f4a0406ed30 con 0x7f49ec00c560 2024-04-15T15:15:29.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.691+0000 7f6fba7fc640 1 -- 172.21.15.70:0/3644526700 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6fbc07ba40 con 0x7f6fc416ba50 2024-04-15T15:15:29.695 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:29.696 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:29.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.693+0000 7f4a00ff9640 1 -- 172.21.15.70:0/3843948987 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f49f4059cc0 con 0x7f49ec00c560 2024-04-15T15:15:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 msgr2=0x7fb8c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 0x7fb8c406ede0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb8e00b3fc0 tx=0x7fb8d0009040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 msgr2=0x7fb8e0071a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc007a50 tx=0x7fb8dc007f10 comp rx=0 tx=0).stop 2024-04-15T15:15:29.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 shutdown_connections 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8e0212ad0 0x7fb8e00ae0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8e0073530 0x7fb8e0212590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb8c406c920 0x7fb8c406ede0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 --2- 172.21.15.70:0/4183144125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8cc00d410 0x7fb8e0071a10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.692+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 >> 172.21.15.70:0/4183144125 conn(0x7fb8e00b1c50 msgr2=0x7fb8e016ae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.693+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 shutdown_connections 2024-04-15T15:15:29.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.693+0000 7fb8e78d9640 1 -- 172.21.15.70:0/4183144125 wait complete. 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.694+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 msgr2=0x7f19d406ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.694+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 0x7f19d406ee00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f19dc036000 tx=0x7f19dc030000 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 msgr2=0x7f19e8073980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f19d80117d0 tx=0x7f19d8011c90 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 shutdown_connections 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f19e8167e90 0x7f19e81f53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f19d406c940 0x7f19d406ee00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19e8204b00 0x7f19e8167670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 --2- 172.21.15.70:0/187813266 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f19d0005090 0x7f19e8073980 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 >> 172.21.15.70:0/187813266 conn(0x7f19e80b1c50 msgr2=0x7f19e80affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 shutdown_connections 2024-04-15T15:15:29.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f19cd7fa640 1 -- 172.21.15.70:0/187813266 wait complete. 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 msgr2=0x7f6fa806ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 0x7f6fa806ee80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6fc40d0690 tx=0x7f6fc0009040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 msgr2=0x7f6fc4099790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc000f80 tx=0x7f6fbc004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 shutdown_connections 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6fc416e9f0 0x7f6fc409d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6fa806c9c0 0x7f6fa806ee80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fc416c3f0 0x7f6fc409d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 --2- 172.21.15.70:0/3644526700 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6fc416ba50 0x7f6fc4099790 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.695+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 >> 172.21.15.70:0/3644526700 conn(0x7f6fc40b5680 msgr2=0x7f6fc4169fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 shutdown_connections 2024-04-15T15:15:29.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f6fcbb32640 1 -- 172.21.15.70:0/3644526700 wait complete. 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 msgr2=0x7f49f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 0x7f49f006edb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4a040b0b30 tx=0x7f49f8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 msgr2=0x7f4a040ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f49f4011a10 tx=0x7f49f4011ed0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 shutdown_connections 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a042092d0 0x7f4a0409ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a04164c80 0x7f4a04208d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f49f006c8f0 0x7f49f006edb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 --2- 172.21.15.70:0/3843948987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49ec00c560 0x7f4a040ae790 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.696+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 >> 172.21.15.70:0/3843948987 conn(0x7f4a040b1c50 msgr2=0x7f4a040af710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.697+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 shutdown_connections 2024-04-15T15:15:29.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.697+0000 7f4a096f7640 1 -- 172.21.15.70:0/3843948987 wait complete. 2024-04-15T15:15:29.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.702+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbfd808b760 con 0x7fbfd80b4240 2024-04-15T15:15:29.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.703+0000 7fbfd57fa640 1 -- 172.21.15.70:0/1874274535 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbfc405cf40 con 0x7fbfd80b4240 2024-04-15T15:15:29.705 INFO:teuthology.orchestra.run.smithi070.stdout:60129542152 2024-04-15T15:15:29.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.706+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7ff474052770 con 0x7ff474101030 2024-04-15T15:15:29.708 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7ff471ffb640 1 -- 172.21.15.70:0/1925134978 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff47c11c070 con 0x7ff474101030 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 msgr2=0x7fbfbc073290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 0x7fbfbc073290 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009af0 tx=0x7fbfcc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 msgr2=0x7fbfd809b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd809b470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbfc400c430 tx=0x7fbfc400c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 shutdown_connections 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfbc070dd0 0x7fbfbc073290 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009af0 tx=0x7fbfcc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfd80cf140 0x7fbfd8072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfd80b4240 0x7fbfd809b470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 --2- 172.21.15.70:0/1874274535 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfd806ef50 0x7fbfd808b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.707+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 >> 172.21.15.70:0/1874274535 conn(0x7fbfd80b1c50 msgr2=0x7fbfd80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.708+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 shutdown_connections 2024-04-15T15:15:29.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.708+0000 7fbfc2ffd640 1 -- 172.21.15.70:0/1874274535 wait complete. 2024-04-15T15:15:29.712 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.5 2024-04-15T15:15:29.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 msgr2=0x7ff46006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 0x7ff46006ee80 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff478006fd0 tx=0x7ff478008040 comp rx=0 tx=0).stop 2024-04-15T15:15:29.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 msgr2=0x7ff4741a28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:29.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 0x7ff4741a28a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0654e0 tx=0x7ff47c069760 comp rx=0 tx=0).stop 2024-04-15T15:15:29.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 shutdown_connections 2024-04-15T15:15:29.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff474101030 0x7ff4741a28a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff46006c9c0 0x7ff46006ee80 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff474100c50 0x7ff474046670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 --2- 172.21.15.70:0/1925134978 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff4740065f0 0x7ff474046130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 >> 172.21.15.70:0/1925134978 conn(0x7ff474009fb0 msgr2=0x7ff474101440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 shutdown_connections 2024-04-15T15:15:29.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:29.711+0000 7ff45f7fe640 1 -- 172.21.15.70:0/1925134978 wait complete. 2024-04-15T15:15:29.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.4 2024-04-15T15:15:29.718 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-04-15T15:15:29.720 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.6 2024-04-15T15:15:29.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.0 2024-04-15T15:15:29.722 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.2 2024-04-15T15:15:29.727 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542152 for osd.7 2024-04-15T15:15:29.728 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574856 for osd.3 2024-04-15T15:15:30.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-15T15:15:30.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-15T15:15:30.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-15T15:15:30.720 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-15T15:15:30.722 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-15T15:15:30.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-15T15:15:30.727 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-15T15:15:30.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-15T15:15:30.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.879+0000 7faae359e640 1 Processor -- start 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.879+0000 7faae359e640 1 -- start start 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae415ba60 0x7faae415e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faae40b4910 0x7faae40b4d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faae40cf3f0 0x7faae40cf880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7faae40cdec0 con 0x7faae40b4910 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faae406ec90 con 0x7faae40cf3f0 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.881+0000 7faae359e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faae406eee0 con 0x7faae415ba60 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faae40b4910 0x7faae40b4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faae40b4910 0x7faae40b4d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36764/0 (socket says 172.21.15.70:36764) 2024-04-15T15:15:30.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 -- 172.21.15.70:0/1881623302 learned_addr learned my addr 172.21.15.70:0/1881623302 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 -- 172.21.15.70:0/1881623302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae415ba60 msgr2=0x7faae415e310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 --2- 172.21.15.70:0/1881623302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae415ba60 0x7faae415e310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 -- 172.21.15.70:0/1881623302 >> v2:172.21.15.96:3300/0 conn(0x7faae40cf3f0 msgr2=0x7faae40cf880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 --2- 172.21.15.70:0/1881623302 >> v2:172.21.15.96:3300/0 conn(0x7faae40cf3f0 0x7faae40cf880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.882+0000 7faae1d9b640 1 -- 172.21.15.70:0/1881623302 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae406f1c0 con 0x7faae40b4910 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.884+0000 7faae1d9b640 1 --2- 172.21.15.70:0/1881623302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae40b4910 0x7faae40b4d70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc007980 tx=0x7faadc030420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb7d47e52e38c99 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.884+0000 7faae159a640 1 -- 172.21.15.70:0/1881623302 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc03e070 con 0x7faae40b4910 2024-04-15T15:15:30.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.884+0000 7faae159a640 1 -- 172.21.15.70:0/1881623302 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc036480 con 0x7faae40b4910 2024-04-15T15:15:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.885+0000 7faae359e640 1 -- 172.21.15.70:0/1881623302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae40b4910 msgr2=0x7faae40b4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.885+0000 7faae359e640 1 --2- 172.21.15.70:0/1881623302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae40b4910 0x7faae40b4d70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faadc007980 tx=0x7faadc030420 comp rx=0 tx=0).stop 2024-04-15T15:15:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 -- 172.21.15.70:0/1881623302 shutdown_connections 2024-04-15T15:15:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 --2- 172.21.15.70:0/1881623302 >> v2:172.21.15.96:3300/0 conn(0x7faae40cf3f0 0x7faae40cf880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 --2- 172.21.15.70:0/1881623302 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae40b4910 0x7faae40b4d70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 --2- 172.21.15.70:0/1881623302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae415ba60 0x7faae415e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 -- 172.21.15.70:0/1881623302 >> 172.21.15.70:0/1881623302 conn(0x7faae40b1c50 msgr2=0x7faae40b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 -- 172.21.15.70:0/1881623302 shutdown_connections 2024-04-15T15:15:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.886+0000 7faae359e640 1 -- 172.21.15.70:0/1881623302 wait complete. 2024-04-15T15:15:30.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.889+0000 7faae359e640 1 Processor -- start 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.889+0000 7faae359e640 1 -- start start 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.890+0000 7faae359e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 0x7faae4201610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.890+0000 7faae359e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 0x7faae4201610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 0x7faae4201610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43040/0 (socket says 172.21.15.70:43040) 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae1d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36770/0 (socket says 172.21.15.70:36770) 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae259c640 1 -- 172.21.15.70:0/2293845517 learned_addr learned my addr 172.21.15.70:0/2293845517 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae359e640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faae4204180 0x7faae41490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.891+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faae415be40 con 0x7faae415ba60 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.892+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faae406f780 con 0x7faae4204180 2024-04-15T15:15:30.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.892+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faae415e850 con 0x7faae40cf3f0 2024-04-15T15:15:30.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.892+0000 7faae2d9d640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faae4204180 0x7faae41490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 -- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 msgr2=0x7faae4201610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 0x7faae4201610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 -- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faae4204180 msgr2=0x7faae41490f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faae4204180 0x7faae41490f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae40afec0 con 0x7faae415ba60 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.893+0000 7faae1d9b640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faadc038000 tx=0x7faadc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.894+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc03e030 con 0x7faae415ba60 2024-04-15T15:15:30.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.894+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae4204660 con 0x7faae415ba60 2024-04-15T15:15:30.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.896+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae4204c20 con 0x7faae415ba60 2024-04-15T15:15:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.897+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc040070 con 0x7faae415ba60 2024-04-15T15:15:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.897+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc036480 con 0x7faae415ba60 2024-04-15T15:15:30.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.897+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faadc042030 con 0x7faae415ba60 2024-04-15T15:15:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.901+0000 7faad2ffd640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 0x7faacc06f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.901+0000 7faae259c640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 0x7faacc06f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.901+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faadc0adaa0 con 0x7faae415ba60 2024-04-15T15:15:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.902+0000 7faae259c640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 0x7faacc06f1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faad8006fd0 tx=0x7faad8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.902+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faac8005290 con 0x7faae415ba60 2024-04-15T15:15:30.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.912+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faadc012070 con 0x7faae415ba60 2024-04-15T15:15:30.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.935+0000 7fa832c90640 1 Processor -- start 2024-04-15T15:15:30.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.935+0000 7fa832c90640 1 -- start start 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa832c90640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa82c0b4d50 0x7fa82c0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa832c90640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa82c0cccf0 0x7fa82c0cdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa832c90640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82c162f30 con 0x7fa82c162b50 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa832c90640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa82c06f2c0 con 0x7fa82c0b4d50 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa832c90640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa82c06f5a0 con 0x7fa82c0cccf0 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7f470a1d3640 1 Processor -- start 2024-04-15T15:15:30.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fdab33c8640 1 Processor -- start 2024-04-15T15:15:30.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fdab33c8640 1 -- start start 2024-04-15T15:15:30.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa831c8e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa82c162b50 0x7fa82c167100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44760/0 (socket says 172.21.15.70:44760) 2024-04-15T15:15:30.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.937+0000 7fa831c8e640 1 -- 172.21.15.70:0/2899449739 learned_addr learned my addr 172.21.15.70:0/2899449739 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:30.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7fd994920640 1 Processor -- start 2024-04-15T15:15:30.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7f470a1d3640 1 -- start start 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f470a1d3640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46fc03c100 0x7f46fc03c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f470a1d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46fc053540 0x7f46fc0539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f47091d1640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f46fc037fe0 0x7f46fc058420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:56048/0 (socket says 172.21.15.70:56048) 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f47089d0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46fc03c100 0x7f46fc03c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f47089d0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f46fc03c100 0x7f46fc03c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:32792/0 (socket says 172.21.15.70:32792) 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fa83248f640 1 --2- 172.21.15.70:0/2899449739 >> v2:172.21.15.96:3300/0 conn(0x7fa82c0cccf0 0x7fa82c0cdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fa83248f640 1 -- 172.21.15.70:0/2899449739 >> v2:172.21.15.70:3300/0 conn(0x7fa82c0b4d50 msgr2=0x7fa82c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fa83248f640 1 --2- 172.21.15.70:0/2899449739 >> v2:172.21.15.70:3300/0 conn(0x7fa82c0b4d50 0x7fa82c0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fa83248f640 1 -- 172.21.15.70:0/2899449739 >> v1:172.21.15.70:6789/0 conn(0x7fa82c162b50 legacy=0x7fa82c167100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fa83248f640 1 -- 172.21.15.70:0/2899449739 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c0b4550 con 0x7fa82c0cccf0 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fdab33c8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdaa4048100 0x7fdaa4006f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fdab33c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa4005f10 con 0x7fdaa40074b0 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fdab33c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdaa4053280 con 0x7fdaa4054560 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7fdab33c8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdaa40e7b90 con 0x7fdaa4048100 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7fdab0963640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdaa40074b0 0x7fdaa4053e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44772/0 (socket says 172.21.15.70:44772) 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7fdab0963640 1 -- 172.21.15.70:0/3435731754 learned_addr learned my addr 172.21.15.70:0/3435731754 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:30.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7fdab1164640 1 --2- 172.21.15.70:0/3435731754 >> v2:172.21.15.96:3300/0 conn(0x7fdaa4048100 0x7fdaa4006f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f4382b0d640 1 Processor -- start 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f4382b0d640 1 -- start start 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7f41d2359640 1 Processor -- start 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.938+0000 7f41d2359640 1 -- start start 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41d2359640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f41cc160390 0x7f41cc0d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41d2359640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc0d1390 con 0x7f41cc0cdc20 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41cb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f41cc0cdc20 0x7f41cc09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44784/0 (socket says 172.21.15.70:44784) 2024-04-15T15:15:30.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41cb7fe640 1 -- 172.21.15.70:0/2916129636 learned_addr learned my addr 172.21.15.70:0/2916129636 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41cc0ccea0 con 0x7f41cc0ce080 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f41cc0cd790 con 0x7f41cc160390 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f41cbfff640 1 --2- 172.21.15.70:0/2916129636 >> v2:172.21.15.70:3300/0 conn(0x7f41cc160390 0x7f41cc0d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7f47099d2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46fc053540 0x7f46fc0539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f47099d2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46fc053540 0x7f46fc0539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43050/0 (socket says 172.21.15.70:43050) 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f47099d2640 1 -- 172.21.15.70:0/851542738 learned_addr learned my addr 172.21.15.70:0/851542738 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46fc0e7990 con 0x7f46fc037fe0 2024-04-15T15:15:30.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f46fc0383c0 con 0x7f46fc03c100 2024-04-15T15:15:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fa83248f640 1 --2- 172.21.15.70:0/2899449739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0cccf0 0x7fa82c0cdf20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa82c06f880 tx=0x7fa82402bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e8b4aa5db8877f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f46fc03cad0 con 0x7f46fc053540 2024-04-15T15:15:30.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f41caffd640 1 -- 172.21.15.70:0/2916129636 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 514420838 0 0) 0x7f41cc0d1390 con 0x7f41cc0cdc20 2024-04-15T15:15:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4fa0e8f640 1 Processor -- start 2024-04-15T15:15:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 912309845 0 0) 0x7fdaa4053280 con 0x7fdaa4054560 2024-04-15T15:15:30.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f41cbfff640 1 -- 172.21.15.70:0/2916129636 >> v1:172.21.15.96:6789/0 conn(0x7f41cc0ce080 legacy=0x7f41cc160770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda98003520 con 0x7fdaa4054560 2024-04-15T15:15:30.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f41cbfff640 1 -- 172.21.15.70:0/2916129636 >> v1:172.21.15.70:6789/0 conn(0x7f41cc0cdc20 legacy=0x7f41cc09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1073365657 0 0) 0x7fdaa4005f10 con 0x7fdaa40074b0 2024-04-15T15:15:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4fa0e8f640 1 -- start start 2024-04-15T15:15:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.939+0000 7fd994920640 1 -- start start 2024-04-15T15:15:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaa4053280 con 0x7fdaa40074b0 2024-04-15T15:15:30.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4382b0d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f437c070460 0x7f437c0d1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4382b0d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c0d0d00 con 0x7f437c1601c0 2024-04-15T15:15:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4382b0d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f437c0d0f80 con 0x7f437c1568b0 2024-04-15T15:15:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f4382b0d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f437c0d1260 con 0x7f437c070460 2024-04-15T15:15:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7f41cbfff640 1 -- 172.21.15.70:0/2916129636 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc0d10b0 con 0x7f41cc160390 2024-04-15T15:15:30.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fa830c8c640 1 -- 172.21.15.70:0/2899449739 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824039070 con 0x7fa82c0cccf0 2024-04-15T15:15:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f43810aa640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f437c1601c0 0x7f437c160620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44796/0 (socket says 172.21.15.70:44796) 2024-04-15T15:15:30.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f43810aa640 1 -- 172.21.15.70:0/2168261824 learned_addr learned my addr 172.21.15.70:0/2168261824 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fa830c8c640 1 -- 172.21.15.70:0/2899449739 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82402cc70 con 0x7fa82c0cccf0 2024-04-15T15:15:30.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fa830c8c640 1 -- 172.21.15.70:0/2899449739 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa824034660 con 0x7fa82c0cccf0 2024-04-15T15:15:30.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd994920640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9900b4fe0 0x7fd99006ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd994920640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 0x7fd99006f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 -- 172.21.15.70:0/2899449739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0cccf0 msgr2=0x7fa82c0cdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 --2- 172.21.15.70:0/2899449739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0cccf0 0x7fa82c0cdf20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa82c06f880 tx=0x7fa82402bec0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd98effd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9900b4fe0 0x7fd99006ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdab1164640 1 -- 172.21.15.70:0/3435731754 >> v1:172.21.15.96:6789/0 conn(0x7fdaa4054560 legacy=0x7fdaa404f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdab1164640 1 -- 172.21.15.70:0/3435731754 >> v1:172.21.15.70:6789/0 conn(0x7fdaa40074b0 legacy=0x7fdaa4053e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.940+0000 7fdab1164640 1 -- 172.21.15.70:0/3435731754 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa40061b0 con 0x7fdaa4048100 2024-04-15T15:15:30.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd98effd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9900b4fe0 0x7fd99006ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:32814/0 (socket says 172.21.15.70:32814) 2024-04-15T15:15:30.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f4fa0e8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c162d70 0x7f4f9c15f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f4fa0e8f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c15f870 con 0x7f4f9c0d3010 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd98ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 0x7fd99006f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd98ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 0x7fd99006f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36786/0 (socket says 172.21.15.70:36786) 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd98effd640 1 -- 172.21.15.70:0/272912341 learned_addr learned my addr 172.21.15.70:0/272912341 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 -- 172.21.15.70:0/2899449739 shutdown_connections 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 --2- 172.21.15.70:0/2899449739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0cccf0 0x7fa82c0cdf20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 --2- 172.21.15.70:0/2899449739 >> v2:172.21.15.70:3300/0 conn(0x7fa82c0b4d50 0x7fa82c0cc7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fa832c90640 1 -- 172.21.15.70:0/2899449739 >> 172.21.15.70:0/2899449739 conn(0x7fa82c0b1c50 msgr2=0x7fa82c0b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9900cfa20 con 0x7fd990162c50 2024-04-15T15:15:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f41cbfff640 1 --2- 172.21.15.70:0/2916129636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc160390 0x7f41cc0d09b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009d90 tx=0x7f41c402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1621dd3f3a01cb36 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f4fa0e8f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9c162290 con 0x7f4f9c15e530 2024-04-15T15:15:30.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f4fa0e8f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9c162570 con 0x7f4f9c162d70 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3739336939 0 0) 0x7f46fc0e7990 con 0x7f46fc037fe0 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46f0003520 con 0x7f46fc037fe0 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.1 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 93029655 0 0) 0x7f46f0003520 con 0x7f46fc037fe0 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46fc053540 msgr2=0x7f46fc0539b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46fc053540 0x7f46fc0539b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 >> v2:172.21.15.96:3300/0 conn(0x7f46fc03c100 msgr2=0x7f46fc03c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 --2- 172.21.15.70:0/851542738 >> v2:172.21.15.96:3300/0 conn(0x7f46fc03c100 0x7f46fc03c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46fc0570a0 con 0x7f46fc037fe0 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.1 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4700001670 con 0x7f46fc037fe0 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 >> v1:172.21.15.96:6789/0 conn(0x7f46fc037fe0 legacy=0x7f46fc058420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46f0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46f0004350 0x7f46f0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0004f40 0x7f46f000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc03cad0 con 0x7f46f0004350 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46f000bf00 con 0x7f46f0004f40 2024-04-15T15:15:30.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f46f000c080 con 0x7f46f00039b0 2024-04-15T15:15:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7f437bfff640 1 --2- 172.21.15.70:0/2168261824 >> v2:172.21.15.96:3300/0 conn(0x7f437c070460 0x7f437c0d1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fa832c90640 1 -- 172.21.15.70:0/2899449739 shutdown_connections 2024-04-15T15:15:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f437b7fe640 1 -- 172.21.15.70:0/2168261824 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203611814 0 0) 0x7f4364001490 con 0x7f437c1601c0 2024-04-15T15:15:30.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f437bfff640 1 -- 172.21.15.70:0/2168261824 >> v1:172.21.15.96:6789/0 conn(0x7f437c1568b0 legacy=0x7f437c06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd990163030 con 0x7fd9900af510 2024-04-15T15:15:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f4f9affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4f9c0d3010 0x7f4f9c15e910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44808/0 (socket says 172.21.15.70:44808) 2024-04-15T15:15:30.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9900b42d0 con 0x7fd9900b4fe0 2024-04-15T15:15:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fa832c90640 1 -- 172.21.15.70:0/2899449739 wait complete. 2024-04-15T15:15:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f4f9affd640 1 -- 172.21.15.70:0/3604441286 learned_addr learned my addr 172.21.15.70:0/3604441286 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c162d70 0x7f4f9c15f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256122510 0 0) 0x7f4f9c162290 con 0x7f4f9c15e530 2024-04-15T15:15:30.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f41caffd640 1 -- 172.21.15.70:0/2916129636 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4039070 con 0x7f41cc160390 2024-04-15T15:15:30.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f41caffd640 1 -- 172.21.15.70:0/2916129636 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41c402cb70 con 0x7f41cc160390 2024-04-15T15:15:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f84003520 con 0x7f4f9c15e530 2024-04-15T15:15:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f47091d1640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46f0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f437bfff640 1 -- 172.21.15.70:0/2168261824 >> v1:172.21.15.70:6789/0 conn(0x7f437c1601c0 legacy=0x7f437c160620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f437bfff640 1 -- 172.21.15.70:0/2168261824 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c0d1540 con 0x7f437c070460 2024-04-15T15:15:30.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7f47089d0640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46f0004350 0x7f46f0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f41caffd640 1 -- 172.21.15.70:0/2916129636 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41c4034440 con 0x7f41cc160390 2024-04-15T15:15:30.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f47091d1640 1 -- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0004f40 msgr2=0x7f46f000b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd98e7fc640 1 -- 172.21.15.70:0/272912341 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211089399 0 0) 0x7fd9900cfa20 con 0x7fd990162c50 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd98effd640 1 -- 172.21.15.70:0/272912341 >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 msgr2=0x7fd99006f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd98effd640 1 --2- 172.21.15.70:0/272912341 >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 0x7fd99006f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd98effd640 1 -- 172.21.15.70:0/272912341 >> v1:172.21.15.96:6789/0 conn(0x7fd990162c50 legacy=0x7fd9900cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fd98effd640 1 -- 172.21.15.70:0/272912341 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9900b4500 con 0x7fd9900b4fe0 2024-04-15T15:15:30.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f47091d1640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46f0004f40 0x7f46f000b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f47091d1640 1 -- 172.21.15.70:0/851542738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46f0004350 msgr2=0x7f46f0004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.941+0000 7fdab1164640 1 --2- 172.21.15.70:0/3435731754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa4006f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80097a0 tx=0x7fdaa802ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1dd2c9b2fc2546fc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa802f830 con 0x7fdaa4048100 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.942+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa802f9d0 con 0x7fdaa4048100 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdaa3fff640 1 -- 172.21.15.70:0/3435731754 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa802fde0 con 0x7fdaa4048100 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdab33c8640 1 -- 172.21.15.70:0/3435731754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 msgr2=0x7fdaa4006f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3435731754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa4006f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80097a0 tx=0x7fdaa802ba10 comp rx=0 tx=0).stop 2024-04-15T15:15:30.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4176107849 0 0) 0x7f4f9c15f870 con 0x7f4f9c0d3010 2024-04-15T15:15:30.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f9c162290 con 0x7f4f9c0d3010 2024-04-15T15:15:30.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.1 v1:172.21.15.96:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2045030319 0 0) 0x7f4f84003520 con 0x7f4f9c15e530 2024-04-15T15:15:30.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c162d70 msgr2=0x7f4f9c15f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c162d70 0x7f4f9c15f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 >> v1:172.21.15.70:6789/0 conn(0x7f4f9c0d3010 legacy=0x7f4f9c15e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdab33c8640 1 -- 172.21.15.70:0/3435731754 shutdown_connections 2024-04-15T15:15:30.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3435731754 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa4006f70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7fdab33c8640 1 -- 172.21.15.70:0/3435731754 >> 172.21.15.70:0/3435731754 conn(0x7fdaa4009fe0 msgr2=0x7fdaa400a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> v1:172.21.15.96:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c162850 con 0x7f4f9c15e530 2024-04-15T15:15:30.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.1 v1:172.21.15.96:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4f8c0015f0 con 0x7f4f9c15e530 2024-04-15T15:15:30.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 >> v1:172.21.15.96:6789/0 conn(0x7f4f9c15e530 legacy=0x7f4f9c0d2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84003b40 0x7f4f84003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84004800 0x7f4f84004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fd98effd640 1 --2- 172.21.15.70:0/272912341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900b4fe0 0x7fd99006ec90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd990074190 tx=0x7fd98402b9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0e4e4a8a32f046f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f840051c0 0x7f4f8400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c162570 con 0x7f4f84004800 2024-04-15T15:15:30.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fa832c90640 1 Processor -- start 2024-04-15T15:15:30.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fdab33c8640 1 -- 172.21.15.70:0/3435731754 shutdown_connections 2024-04-15T15:15:30.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f84014740 con 0x7f4f84003b40 2024-04-15T15:15:30.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f84014940 con 0x7f4f840051c0 2024-04-15T15:15:30.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fa832c90640 1 -- start start 2024-04-15T15:15:30.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc160390 msgr2=0x7f41cc0d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 --2- 172.21.15.70:0/2916129636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc160390 0x7f41cc0d09b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f41c4009d90 tx=0x7f41c402be40 comp rx=0 tx=0).stop 2024-04-15T15:15:30.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fd98e7fc640 1 -- 172.21.15.70:0/272912341 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984034070 con 0x7fd9900b4fe0 2024-04-15T15:15:30.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fd98e7fc640 1 -- 172.21.15.70:0/272912341 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd984030880 con 0x7fd9900b4fe0 2024-04-15T15:15:30.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fd98e7fc640 1 -- 172.21.15.70:0/272912341 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984030c50 con 0x7fd9900b4fe0 2024-04-15T15:15:30.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 shutdown_connections 2024-04-15T15:15:30.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 --2- 172.21.15.70:0/2916129636 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc160390 0x7f41cc0d09b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 >> 172.21.15.70:0/2916129636 conn(0x7f41cc0b3ec0 msgr2=0x7f41cc0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f840051c0 0x7f4f8400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 shutdown_connections 2024-04-15T15:15:30.976 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84004800 0x7f4f84004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f41d2359640 1 -- 172.21.15.70:0/2916129636 wait complete. 2024-04-15T15:15:30.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f437bfff640 1 --2- 172.21.15.70:0/2168261824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c0d1d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f436c0097a0 tx=0x7f436c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0e262b779f7322c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9b7fe640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84003b40 0x7f4f84003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 -- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f840051c0 msgr2=0x7f4f8400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f840051c0 0x7f4f8400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 -- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84003b40 msgr2=0x7f4f84003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84003b40 0x7f4f84003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f47091d1640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46f0004350 0x7f46f0004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.943+0000 7f47091d1640 1 -- 172.21.15.70:0/851542738 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46fc0570a0 con 0x7f46f00039b0 2024-04-15T15:15:30.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f47089d0640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46f0004350 0x7f46f0004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f47091d1640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46f0003dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f470000c340 tx=0x7f46f0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a99fa7b4bafa92db server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700007d90 con 0x7f46f00039b0 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700030070 con 0x7f46f00039b0 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f46fbfff640 1 -- 172.21.15.70:0/851542738 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700007d90 con 0x7f46f00039b0 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9affd640 1 -- 172.21.15.70:0/3604441286 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c162850 con 0x7f4f84004800 2024-04-15T15:15:30.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7f4f9b7fe640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84003b40 0x7f4f84003f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900b4fe0 msgr2=0x7fd99006ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7fd994920640 1 --2- 172.21.15.70:0/272912341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900b4fe0 0x7fd99006ec90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd990074190 tx=0x7fd98402b9c0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f840051c0 0x7f4f8400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f4f9affd640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84004800 0x7f4f84004c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400c340 tx=0x7f4f84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3adc8f5fe155251 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7fa832c90640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f437b7fe640 1 -- 172.21.15.70:0/2168261824 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c035020 con 0x7f437c070460 2024-04-15T15:15:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7f437b7fe640 1 -- 172.21.15.70:0/2168261824 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f436c031830 con 0x7f437c070460 2024-04-15T15:15:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.945+0000 7fa832c90640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 0x7fa82c09d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa831c8e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 msgr2=0x7f46f0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.944+0000 7fdab33c8640 1 -- 172.21.15.70:0/3435731754 wait complete. 2024-04-15T15:15:30.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 --2- 172.21.15.70:0/851542738 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46f0003dd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f470000c340 tx=0x7f46f0005830 comp rx=0 tx=0).stop 2024-04-15T15:15:30.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f47099d2640 1 -- 172.21.15.70:0/851542738 reap_dead start 2024-04-15T15:15:30.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fdab33c8640 1 Processor -- start 2024-04-15T15:15:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f437b7fe640 1 -- 172.21.15.70:0/2168261824 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f436c031c00 con 0x7f437c070460 2024-04-15T15:15:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa831c8e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:32844/0 (socket says 172.21.15.70:32844) 2024-04-15T15:15:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa83148d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 0x7fa82c09d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa83148d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 0x7fa82c09d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43092/0 (socket says 172.21.15.70:43092) 2024-04-15T15:15:30.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa831c8e640 1 -- 172.21.15.70:0/1011371442 learned_addr learned my addr 172.21.15.70:0/1011371442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 -- 172.21.15.70:0/2168261824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 msgr2=0x7f437c0d1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2168261824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c0d1d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f436c0097a0 tx=0x7f436c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 0x7fa82c09d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 shutdown_connections 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 >> 172.21.15.70:0/851542738 conn(0x7f46fc058ef0 msgr2=0x7f46fc059300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c0af5d0 con 0x7fa82c162b50 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa82c06ef50 con 0x7fa82c0b4d50 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa82c0af750 con 0x7fa82c0cccf0 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 -- 172.21.15.70:0/2168261824 shutdown_connections 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2168261824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c0d1d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 -- 172.21.15.70:0/2168261824 >> 172.21.15.70:0/2168261824 conn(0x7f437c067d50 msgr2=0x7f437c0b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 -- 172.21.15.70:0/2168261824 shutdown_connections 2024-04-15T15:15:30.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fdab33c8640 1 -- start start 2024-04-15T15:15:30.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4382b0d640 1 -- 172.21.15.70:0/2168261824 wait complete. 2024-04-15T15:15:30.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94002020 con 0x7f4f84004800 2024-04-15T15:15:30.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f94030070 con 0x7f4f84004800 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f4f9a7fc640 1 -- 172.21.15.70:0/3604441286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f94035470 con 0x7f4f84004800 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 shutdown_connections 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7f470a1d3640 1 -- 172.21.15.70:0/851542738 wait complete. 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 shutdown_connections 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fd994920640 1 --2- 172.21.15.70:0/272912341 >> v2:172.21.15.70:3300/0 conn(0x7fd9900af510 0x7fd99006f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fd994920640 1 --2- 172.21.15.70:0/272912341 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900b4fe0 0x7fd99006ec90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 >> 172.21.15.70:0/272912341 conn(0x7fd9900b1c50 msgr2=0x7fd9900b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f41d2359640 1 Processor -- start 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/3604441286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84004800 msgr2=0x7f4f84004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/3604441286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84004800 0x7f4f84004c80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4f9400c340 tx=0x7f4f84005830 comp rx=0 tx=0).stop 2024-04-15T15:15:30.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.946+0000 7fa83248f640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 0x7fa82c09d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa831c8e640 1 -- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 msgr2=0x7fa82c09d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa831c8e640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 0x7fa82c09d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4f9bfff640 1 -- 172.21.15.70:0/3604441286 reap_dead start 2024-04-15T15:15:30.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa831c8e640 1 -- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 msgr2=0x7fa82c09d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa831c8e640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 0x7fa82c09d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa831c8e640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa82c161100 con 0x7fa82c0b4d50 2024-04-15T15:15:30.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f41d2359640 1 -- start start 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/3604441286 shutdown_connections 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/3604441286 >> 172.21.15.70:0/3604441286 conn(0x7f4f9c09d780 msgr2=0x7f4f9c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/3604441286 shutdown_connections 2024-04-15T15:15:30.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fa83248f640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 0x7fa82c09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:30.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/3604441286 wait complete. 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdaa4054560 0x7fdaa404e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab1164640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab0963640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab1164640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36838/0 (socket says 172.21.15.70:36838) 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4006770 con 0x7fdaa40074b0 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa4006ae0 con 0x7fdaa4048100 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab1164640 1 -- 172.21.15.70:0/3533078959 learned_addr learned my addr 172.21.15.70:0/3533078959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdaa4050040 con 0x7fdaa4054560 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7f4382b0d640 1 Processor -- start 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.947+0000 7fdab0963640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:32858/0 (socket says 172.21.15.70:32858) 2024-04-15T15:15:30.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7f41d2359640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 0x7f41cc1f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7f4382b0d640 1 -- start start 2024-04-15T15:15:30.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7f41d2359640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 0x7f41cc0b5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1965640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdaa4054560 0x7fdaa404e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 -- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdaa4054560 msgr2=0x7fdaa404e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d2359640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 0x7f41cc0b5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:30.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f470a1d3640 1 Processor -- start 2024-04-15T15:15:30.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdaa4054560 0x7fdaa404e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 -- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 msgr2=0x7fdaa404dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa40d64e0 con 0x7fdaa40074b0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab0963640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.948+0000 7fdab1164640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80098c0 tx=0x7fdaa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037bc0 con 0x7fdaa40074b0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8047070 con 0x7fdaa40074b0 2024-04-15T15:15:30.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f470a1d3640 1 -- start start 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 0x7f41cc0b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41cb7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 0x7f41cc0b5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43110/0 (socket says 172.21.15.70:43110) 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41cb7fe640 1 -- 172.21.15.70:0/1848138223 learned_addr learned my addr 172.21.15.70:0/1848138223 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc15dbd0 con 0x7f41cc0cdc20 2024-04-15T15:15:30.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d08f6640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 0x7f41cc0b5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41cc0cd420 con 0x7f41cc160390 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f41cc09b0e0 con 0x7f41cc0ce080 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7fa831c8e640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c480 tx=0x7fa81c00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa4036cc0 con 0x7fdaa40074b0 2024-04-15T15:15:30.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f4fa0e8f640 1 Processor -- start 2024-04-15T15:15:31.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d08f6640 1 -- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 msgr2=0x7f41cc0b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d08f6640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 0x7f41cc0b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f4fa0e8f640 1 -- start start 2024-04-15T15:15:31.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7f41d08f6640 1 -- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 msgr2=0x7f41cc1f1a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-15T15:15:31.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41cbfff640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 0x7f41cc1f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.949+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8042480 con 0x7fdaa40074b0 2024-04-15T15:15:31.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa404bfd0 con 0x7fdaa40074b0 2024-04-15T15:15:31.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f4382b0d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c14fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f470a1d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41d08f6640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 0x7f41cc1f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41d08f6640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41cc0b6400 con 0x7f41cc160390 2024-04-15T15:15:31.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f4382b0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 0x7f437c0b2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41cbfff640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 0x7f41cc1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:31.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f470a1d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46fc0dc9e0 0x7f46fc03a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41d08f6640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 0x7f41cc0b5dd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00c400 tx=0x7f41bc00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f4382b0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 0x7f437c0b0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f47091d1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f47091d1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43120/0 (socket says 172.21.15.70:43120) 2024-04-15T15:15:31.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f437bfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 0x7f437c0b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f47091d1640 1 -- 172.21.15.70:0/3002440109 learned_addr learned my addr 172.21.15.70:0/3002440109 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f47089d0640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46fc0dc9e0 0x7f46fc03a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f437bfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 0x7f437c0b2900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43130/0 (socket says 172.21.15.70:43130) 2024-04-15T15:15:31.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f437bfff640 1 -- 172.21.15.70:0/2921542772 learned_addr learned my addr 172.21.15.70:0/2921542772 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f437c0b56c0 con 0x7f437c1601c0 2024-04-15T15:15:31.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f437c0b5540 con 0x7f437c070460 2024-04-15T15:15:31.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f437c0b5840 con 0x7f437c1568b0 2024-04-15T15:15:31.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c017070 con 0x7fa82c0b4d50 2024-04-15T15:15:31.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa81c005ce0 con 0x7fa82c0b4d50 2024-04-15T15:15:31.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa81c0072d0 con 0x7fa82c0b4d50 2024-04-15T15:15:31.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46fc0e3d60 0x7f46fc0ea3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc057940 con 0x7f46fc0e3d60 2024-04-15T15:15:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4fa0e8f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0cdc30 0x7f4f9c0b2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4fa0e8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 0x7f4f9c0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4fa0e8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 0x7f4f9c146800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa82c153c80 con 0x7fa82c0b4d50 2024-04-15T15:15:31.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa82c09c2c0 con 0x7fa82c0b4d50 2024-04-15T15:15:31.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa400ad00 con 0x7fdaa40074b0 2024-04-15T15:15:31.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f43808a9640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c14fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4f9affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 0x7f4f9c0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f43810aa640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 0x7f437c0b0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 shutdown_connections 2024-04-15T15:15:31.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f47099d2640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46fc0e3d60 0x7f46fc0ea3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc057660 con 0x7f46fc0dc9e0 2024-04-15T15:15:31.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f46fc057c20 con 0x7f46f00039b0 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.950+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc017070 con 0x7f41cc160390 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41bc0035f0 con 0x7f41cc160390 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41bc0039e0 con 0x7f41cc160390 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4fa0e8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c15f3f0 con 0x7f4f9c158dd0 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4f9affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 0x7f4f9c0b7000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36870/0 (socket says 172.21.15.70:36870) 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.951+0000 7f4f9affd640 1 -- 172.21.15.70:0/4191614614 learned_addr learned my addr 172.21.15.70:0/4191614614 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 0x7f4f9c146800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9c15f270 con 0x7f4f9c0cdc30 2024-04-15T15:15:31.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9c15f570 con 0x7f4f9c0ce010 2024-04-15T15:15:31.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41cc0b66e0 con 0x7f41cc160390 2024-04-15T15:15:31.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43808a9640 1 -- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 msgr2=0x7f437c0b2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41cc0b6ca0 con 0x7f41cc160390 2024-04-15T15:15:31.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9b7fe640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0cdc30 0x7f4f9c0b2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43808a9640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 0x7f437c0b2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7fd994920640 1 -- 172.21.15.70:0/272912341 wait complete. 2024-04-15T15:15:31.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43808a9640 1 -- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 msgr2=0x7f437c0b0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43808a9640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 0x7f437c0b0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43808a9640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f437c0b3230 con 0x7f437c070460 2024-04-15T15:15:31.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f43810aa640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 0x7f437c0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:31.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdaa8040050 con 0x7fdaa40074b0 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 -- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0cdc30 msgr2=0x7f4f9c0b2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0cdc30 0x7f4f9c0b2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 -- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 msgr2=0x7f4f9c0b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 0x7f4f9c0b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7f4f9bfff640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f9c1592d0 con 0x7f4f9c0ce010 2024-04-15T15:15:31.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.952+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa814005290 con 0x7fa82c0b4d50 2024-04-15T15:15:31.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f47091d1640 1 -- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46fc0dc9e0 msgr2=0x7f46fc03a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f47091d1640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46fc0dc9e0 0x7f46fc03a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f47091d1640 1 -- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46fc0e3d60 msgr2=0x7f46fc0ea3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7fdaa1ffb640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 0x7fda9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f47091d1640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46fc0e3d60 0x7f46fc0ea3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f47091d1640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46fc0e32c0 con 0x7f46f00039b0 2024-04-15T15:15:31.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdaa80ae880 con 0x7fdaa40074b0 2024-04-15T15:15:31.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f43808a9640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c14fdb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f437400c440 tx=0x7f437400c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f4f9bfff640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 0x7f4f9c146800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0b6f60 tx=0x7f4f90011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90009650 con 0x7f4f9c0ce010 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f9001c070 con 0x7f4f9c0ce010 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.953+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f9000fc40 con 0x7f4f9c0ce010 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9c0b1b70 con 0x7f4f9c0ce010 2024-04-15T15:15:31.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f9c0b2130 con 0x7f4f9c0ce010 2024-04-15T15:15:31.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41bc01a070 con 0x7f41cc160390 2024-04-15T15:15:31.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7fd994920640 1 Processor -- start 2024-04-15T15:15:31.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7f47091d1640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f470000c460 tx=0x7f4700000fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7fdab0963640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 0x7fda9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700045070 con 0x7f46f00039b0 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4700005e30 con 0x7f46f00039b0 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46fc03aed0 con 0x7f46f00039b0 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4700045070 con 0x7f46f00039b0 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7fd994920640 1 -- start start 2024-04-15T15:15:31.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f9c072f30 con 0x7f4f9c0ce010 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa81c01a070 con 0x7fa82c0b4d50 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900af510 0x7fd99015dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9900b4fe0 0x7fd990156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900b4ac0 con 0x7fd9900b4fe0 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9900cf7c0 con 0x7fd9900af510 2024-04-15T15:15:31.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd994920640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd9900cfd00 con 0x7fd990162c50 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46fc0e18b0 con 0x7f46f00039b0 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7fa8227fc640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 0x7fa81006ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.954+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41cc0690f0 con 0x7f41cc160390 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f41c8ff9640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 0x7f41b806ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.955+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f41bc08b110 con 0x7f41cc160390 2024-04-15T15:15:31.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7f41cbfff640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 0x7f41b806ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa81c08b520 con 0x7fa82c0b4d50 2024-04-15T15:15:31.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd98ffff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd98ffff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43146/0 (socket says 172.21.15.70:43146) 2024-04-15T15:15:31.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd98ffff640 1 -- 172.21.15.70:0/1709939347 learned_addr learned my addr 172.21.15.70:0/1709939347 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f90012040 con 0x7f4f9c0ce010 2024-04-15T15:15:31.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fa83148d640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 0x7fa81006ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd98f7fe640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900af510 0x7fd99015dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7fd98effd640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9900b4fe0 0x7fd990156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 -- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900af510 msgr2=0x7fd99015dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7f4f83fff640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 0x7f4f8806eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7f41cbfff640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 0x7f41b806ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41c4036000 tx=0x7f41c4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.956+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46fc005710 con 0x7f46f00039b0 2024-04-15T15:15:31.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900af510 0x7fd99015dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fa83148d640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 0x7fa81006ee80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0ce630 tx=0x7fa828009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7f4f9b7fe640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 0x7f4f8806eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4f9008baf0 con 0x7f4f9c0ce010 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 -- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9900b4fe0 msgr2=0x7fd990156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9900b4fe0 0x7fd990156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 -- 172.21.15.70:0/1709939347 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99009b170 con 0x7fd990162c50 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fd98ffff640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd98800efc0 tx=0x7fd98800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.957+0000 7fdab0963640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 0x7fda9006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4180fb0 tx=0x7fda9c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7f4f9b7fe640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 0x7f4f8806eeb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c000fd0 tx=0x7f4f8c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374016020 con 0x7f437c070460 2024-04-15T15:15:31.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4700048070 con 0x7f46f00039b0 2024-04-15T15:15:31.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa81c058fa0 con 0x7fa82c0b4d50 2024-04-15T15:15:31.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f437c157090 con 0x7f437c070460 2024-04-15T15:15:31.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.958+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f437c0b3510 con 0x7f437c070460 2024-04-15T15:15:31.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.959+0000 7f46f9ffb640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 0x7f46e806f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.959+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f470002c960 con 0x7f46f00039b0 2024-04-15T15:15:31.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.959+0000 7f47089d0640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 0x7f46e806f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.959+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f437c1577a0 con 0x7f437c070460 2024-04-15T15:15:31.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.960+0000 7f47089d0640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 0x7f46e806f1e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0531a0 tx=0x7f46f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.960+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdaa40d64e0 con 0x7fdaa40074b0 2024-04-15T15:15:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.962+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988018020 con 0x7fd990162c50 2024-04-15T15:15:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.962+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd99015e1b0 con 0x7fd990162c50 2024-04-15T15:15:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.962+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd99015e770 con 0x7fd990162c50 2024-04-15T15:15:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.962+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f9c072f30 con 0x7f4f9c0ce010 2024-04-15T15:15:31.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.963+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4374005080 con 0x7f437c070460 2024-04-15T15:15:31.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.963+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4374016020 con 0x7f437c070460 2024-04-15T15:15:31.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.963+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41bc058c90 con 0x7f41cc160390 2024-04-15T15:15:31.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.965+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4374019020 con 0x7f437c070460 2024-04-15T15:15:31.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.965+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd99004fd50 con 0x7fd990162c50 2024-04-15T15:15:31.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.965+0000 7f43797fa640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 0x7f435c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.965+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4374057520 con 0x7f437c070460 2024-04-15T15:15:31.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.966+0000 7f437bfff640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 0x7f435c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.966+0000 7f437bfff640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 0x7f435c06efb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f437c159700 tx=0x7f436c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.966+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98800cd30 con 0x7fd990162c50 2024-04-15T15:15:31.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.966+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988014680 con 0x7fd990162c50 2024-04-15T15:15:31.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.967+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f470007a680 con 0x7f46f00039b0 2024-04-15T15:15:31.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.968+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd988005d10 con 0x7fd990162c50 2024-04-15T15:15:31.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.969+0000 7fd977fff640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 0x7fd97c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.969+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd98808d430 con 0x7fd990162c50 2024-04-15T15:15:31.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.969+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4374020020 con 0x7f437c070460 2024-04-15T15:15:31.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.969+0000 7fd98f7fe640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 0x7fd97c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.970+0000 7fd98f7fe640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 0x7fd97c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd980009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:30.971+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd988022030 con 0x7fd990162c50 2024-04-15T15:15:31.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.319+0000 7faae359e640 1 -- 172.21.15.70:0/2293845517 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7faac8005850 con 0x7faae415ba60 2024-04-15T15:15:31.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.319+0000 7faad2ffd640 1 -- 172.21.15.70:0/2293845517 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7faadc07bc60 con 0x7faae415ba60 2024-04-15T15:15:31.321 INFO:teuthology.orchestra.run.smithi070.stdout:55834574856 2024-04-15T15:15:31.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 msgr2=0x7faacc06f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 0x7faacc06f1e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7faad8006fd0 tx=0x7faad8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 msgr2=0x7faae409c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faadc038000 tx=0x7faadc004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 shutdown_connections 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faae4204180 0x7faae41490f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faacc06cd20 0x7faacc06f1e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faae415ba60 0x7faae409c540 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.323+0000 7faad0ff9640 1 --2- 172.21.15.70:0/2293845517 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faae40cf3f0 0x7faae4201610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.324+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 >> 172.21.15.70:0/2293845517 conn(0x7faae40b1c50 msgr2=0x7faae415a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.324+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 shutdown_connections 2024-04-15T15:15:31.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.324+0000 7faad0ff9640 1 -- 172.21.15.70:0/2293845517 wait complete. 2024-04-15T15:15:31.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2024-04-15T15:15:31.343 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.356+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa814005850 con 0x7fa82c0b4d50 2024-04-15T15:15:31.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.357+0000 7fa8227fc640 1 -- 172.21.15.70:0/1011371442 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa81c05d2a0 con 0x7fa82c0b4d50 2024-04-15T15:15:31.359 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 msgr2=0x7fa81006ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 0x7fa81006ee80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0ce630 tx=0x7fa828009110 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 msgr2=0x7fa82c09cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00c480 tx=0x7fa81c00c880 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 shutdown_connections 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.359+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81006c9c0 0x7fa81006ee80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa82c162b50 0x7fa82c09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa82c0cccf0 0x7fa82c09d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 --2- 172.21.15.70:0/1011371442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa82c0b4d50 0x7fa82c09cda0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 >> 172.21.15.70:0/1011371442 conn(0x7fa82c0b1c50 msgr2=0x7fa82c15ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 shutdown_connections 2024-04-15T15:15:31.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.360+0000 7fa832c90640 1 -- 172.21.15.70:0/1011371442 wait complete. 2024-04-15T15:15:31.373 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.0 2024-04-15T15:15:31.373 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.383+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fdaa4052850 con 0x7fdaa40074b0 2024-04-15T15:15:31.385 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.383+0000 7fdaa1ffb640 1 -- 172.21.15.70:0/3533078959 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdaa8033030 con 0x7fdaa40074b0 2024-04-15T15:15:31.385 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.385+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 msgr2=0x7fda9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.385+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 0x7fda9006edb0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4180fb0 tx=0x7fda9c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 msgr2=0x7fdaa40d5fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80098c0 tx=0x7fdaa80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 shutdown_connections 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdaa4054560 0x7fdaa404e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdaa4048100 0x7fdaa404dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fda9006c8f0 0x7fda9006edb0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 --2- 172.21.15.70:0/3533078959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdaa40074b0 0x7fdaa40d5fa0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 >> 172.21.15.70:0/3533078959 conn(0x7fdaa4009fe0 msgr2=0x7fdaa40eb1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 shutdown_connections 2024-04-15T15:15:31.388 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.386+0000 7fdab33c8640 1 -- 172.21.15.70:0/3533078959 wait complete. 2024-04-15T15:15:31.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.389+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f437c04fd50 con 0x7f437c070460 2024-04-15T15:15:31.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.389+0000 7f43797fa640 1 -- 172.21.15.70:0/2921542772 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4374020020 con 0x7f437c070460 2024-04-15T15:15:31.391 INFO:teuthology.orchestra.run.smithi070.stdout:60129542154 2024-04-15T15:15:31.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.390+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f41cc0b6ea0 con 0x7f41cc160390 2024-04-15T15:15:31.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.390+0000 7f41c8ff9640 1 -- 172.21.15.70:0/1848138223 <== mon.1 v2:172.21.15.96:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41bc05cf90 con 0x7f41cc160390 2024-04-15T15:15:31.393 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:31.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.391+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4f9c04fd50 con 0x7f4f9c0ce010 2024-04-15T15:15:31.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.391+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f46fc00e140 con 0x7f46f00039b0 2024-04-15T15:15:31.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 msgr2=0x7f435c06efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 0x7f435c06efb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f437c159700 tx=0x7f436c03d040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4f83fff640 1 -- 172.21.15.70:0/4191614614 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4f90059c50 con 0x7f4f9c0ce010 2024-04-15T15:15:31.394 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f46f9ffb640 1 -- 172.21.15.70:0/3002440109 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f470007e980 con 0x7f46f00039b0 2024-04-15T15:15:31.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 msgr2=0x7f437c14fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c14fdb0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f437400c440 tx=0x7f437400c900 comp rx=0 tx=0).stop 2024-04-15T15:15:31.395 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 shutdown_connections 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f435c06caf0 0x7f435c06efb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f437c1601c0 0x7f437c0b0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f437c1568b0 0x7f437c0b2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 --2- 172.21.15.70:0/2921542772 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f437c070460 0x7f437c14fdb0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 >> 172.21.15.70:0/2921542772 conn(0x7f437c067d50 msgr2=0x7f437c152030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 shutdown_connections 2024-04-15T15:15:31.397 INFO:teuthology.orchestra.run.smithi070.stdout:55834574858 2024-04-15T15:15:31.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.392+0000 7f4382b0d640 1 -- 172.21.15.70:0/2921542772 wait complete. 2024-04-15T15:15:31.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.393+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 msgr2=0x7f41b806ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.393+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 0x7f41b806ee00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f41c4036000 tx=0x7f41c4030000 comp rx=0 tx=0).stop 2024-04-15T15:15:31.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.393+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 msgr2=0x7f41cc0b5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.393+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 0x7f41cc0b5dd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f41bc00c400 tx=0x7f41bc00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 shutdown_connections 2024-04-15T15:15:31.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f41cc160390 0x7f41cc0b5dd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41cc0ce080 0x7f41cc0b5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41b806c940 0x7f41b806ee00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 --2- 172.21.15.70:0/1848138223 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41cc0cdc20 0x7f41cc1f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 >> 172.21.15.70:0/1848138223 conn(0x7f41cc0b3ec0 msgr2=0x7f41cc0cede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 shutdown_connections 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.394+0000 7f41d2359640 1 -- 172.21.15.70:0/1848138223 wait complete. 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 msgr2=0x7f4f8806eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 0x7f4f8806eeb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c000fd0 tx=0x7f4f8c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.400 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 msgr2=0x7f4f9c146800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 0x7f4f9c146800 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f9c0b6f60 tx=0x7f4f90011d70 comp rx=0 tx=0).stop 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 shutdown_connections 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9c0ce010 0x7f4f9c146800 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f8806c9f0 0x7f4f8806eeb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f9c158dd0 0x7f4f9c0b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 --2- 172.21.15.70:0/4191614614 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9c0cdc30 0x7f4f9c0b2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 >> 172.21.15.70:0/4191614614 conn(0x7f4f9c09d780 msgr2=0x7f4f9c0d4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.401 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fd9900cd850 con 0x7fd990162c50 2024-04-15T15:15:31.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 shutdown_connections 2024-04-15T15:15:31.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f4fa0e8f640 1 -- 172.21.15.70:0/4191614614 wait complete. 2024-04-15T15:15:31.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7fd977fff640 1 -- 172.21.15.70:0/1709939347 <== mon.2 v2:172.21.15.96:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd988010040 con 0x7fd990162c50 2024-04-15T15:15:31.402 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 msgr2=0x7f46e806f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 0x7f46e806f1e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0531a0 tx=0x7f46f4008040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 msgr2=0x7f46fc03b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f470000c460 tx=0x7f4700000fe0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 shutdown_connections 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f46e806cd20 0x7f46e806f1e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f46fc0e3d60 0x7f46fc0ea3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f46fc0dc9e0 0x7f46fc03a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 --2- 172.21.15.70:0/3002440109 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f46f00039b0 0x7f46fc03b200 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.395+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 >> 172.21.15.70:0/3002440109 conn(0x7f46fc058ef0 msgr2=0x7f46fc04db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.403 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.396+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 shutdown_connections 2024-04-15T15:15:31.404 INFO:teuthology.orchestra.run.smithi070.stdout:55834574857 2024-04-15T15:15:31.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.396+0000 7f470a1d3640 1 -- 172.21.15.70:0/3002440109 wait complete. 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 msgr2=0x7fd97c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 0x7fd97c06edb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd980000f60 tx=0x7fd980009040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 msgr2=0x7fd99015d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd98800efc0 tx=0x7fd98800c490 comp rx=0 tx=0).stop 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 shutdown_connections 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd990162c50 0x7fd99015d3c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.405 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd97c06c8f0 0x7fd97c06edb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9900b4fe0 0x7fd990156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 --2- 172.21.15.70:0/1709939347 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9900af510 0x7fd99015dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.398+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 >> 172.21.15.70:0/1709939347 conn(0x7fd9900b1c50 msgr2=0x7fd9900b2950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.399+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 shutdown_connections 2024-04-15T15:15:31.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.399+0000 7fd994920640 1 -- 172.21.15.70:0/1709939347 wait complete. 2024-04-15T15:15:31.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2024-04-15T15:15:31.407 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.7 2024-04-15T15:15:31.412 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.414 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574858 for osd.1 2024-04-15T15:15:31.414 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.415 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2024-04-15T15:15:31.415 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2024-04-15T15:15:31.417 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.417 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2024-04-15T15:15:31.417 DEBUG:teuthology.parallel:result is None 2024-04-15T15:15:31.418 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:15:31.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:31.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.557+0000 7fef4257c640 1 Processor -- start 2024-04-15T15:15:31.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.557+0000 7fef4257c640 1 -- start start 2024-04-15T15:15:31.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef4257c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fef3c0b3160 0x7fef3c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef4257c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef3c15cb50 con 0x7fef3c0ae280 2024-04-15T15:15:31.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef4257c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef3c15bfd0 con 0x7fef3c0d0690 2024-04-15T15:15:31.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef4257c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fef3c15c2b0 con 0x7fef3c0b3160 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef40b19640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fef3c0ae280 0x7fef3c0ae710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44812/0 (socket says 172.21.15.70:44812) 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef40b19640 1 -- 172.21.15.70:0/2650474395 learned_addr learned my addr 172.21.15.70:0/2650474395 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3b7fe640 1 --2- 172.21.15.70:0/2650474395 >> v2:172.21.15.96:3300/0 conn(0x7fef3c0b3160 0x7fef3c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3affd640 1 -- 172.21.15.70:0/2650474395 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2275712086 0 0) 0x7fef3c15cb50 con 0x7fef3c0ae280 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3affd640 1 -- 172.21.15.70:0/2650474395 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef24003520 con 0x7fef3c0ae280 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3b7fe640 1 -- 172.21.15.70:0/2650474395 >> v1:172.21.15.96:6789/0 conn(0x7fef3c0d0690 legacy=0x7fef3c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3b7fe640 1 -- 172.21.15.70:0/2650474395 >> v1:172.21.15.70:6789/0 conn(0x7fef3c0ae280 legacy=0x7fef3c0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.558+0000 7fef3b7fe640 1 -- 172.21.15.70:0/2650474395 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c15c590 con 0x7fef3c0b3160 2024-04-15T15:15:31.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.559+0000 7fef3b7fe640 1 --2- 172.21.15.70:0/2650474395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0b3160 0x7fef3c15b690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef300097a0 tx=0x7fef3002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9343a17fad334ff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.559+0000 7fef3affd640 1 -- 172.21.15.70:0/2650474395 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30034070 con 0x7fef3c0b3160 2024-04-15T15:15:31.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.559+0000 7fef3affd640 1 -- 172.21.15.70:0/2650474395 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef30030610 con 0x7fef3c0b3160 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.559+0000 7fef3affd640 1 -- 172.21.15.70:0/2650474395 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300309e0 con 0x7fef3c0b3160 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 -- 172.21.15.70:0/2650474395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0b3160 msgr2=0x7fef3c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 --2- 172.21.15.70:0/2650474395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0b3160 0x7fef3c15b690 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fef300097a0 tx=0x7fef3002b960 comp rx=0 tx=0).stop 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 -- 172.21.15.70:0/2650474395 shutdown_connections 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 --2- 172.21.15.70:0/2650474395 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0b3160 0x7fef3c15b690 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 -- 172.21.15.70:0/2650474395 >> 172.21.15.70:0/2650474395 conn(0x7fef3c06be00 msgr2=0x7fef3c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 -- 172.21.15.70:0/2650474395 shutdown_connections 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.560+0000 7fef4257c640 1 -- 172.21.15.70:0/2650474395 wait complete. 2024-04-15T15:15:31.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.561+0000 7fef4257c640 1 Processor -- start 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.561+0000 7fef4257c640 1 -- start start 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0ae280 0x7fef3c15f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fef3c0d0690 0x7fef3c0b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c15bbd0 con 0x7fef3c0b3160 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fef3c0b4fb0 con 0x7fef3c0ae280 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef4257c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fef3c15bd50 con 0x7fef3c0d0690 2024-04-15T15:15:31.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36892/0 (socket says 172.21.15.70:36892) 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 -- 172.21.15.70:0/4159342547 learned_addr learned my addr 172.21.15.70:0/4159342547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3bfff640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0ae280 0x7fef3c15f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef40b19640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fef3c0d0690 0x7fef3c0b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 -- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fef3c0d0690 msgr2=0x7fef3c0b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fef3c0d0690 0x7fef3c0b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 -- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0ae280 msgr2=0x7fef3c15f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0ae280 0x7fef3c15f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 -- 172.21.15.70:0/4159342547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3c1552a0 con 0x7fef3c0b3160 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.562+0000 7fef3b7fe640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef300098c0 tx=0x7fef300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.563+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef30034040 con 0x7fef3c0b3160 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.563+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef3003aaa0 con 0x7fef3c0b3160 2024-04-15T15:15:31.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.563+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3c155580 con 0x7fef3c0b3160 2024-04-15T15:15:31.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.563+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef300443f0 con 0x7fef3c0b3160 2024-04-15T15:15:31.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.563+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3c155b40 con 0x7fef3c0b3160 2024-04-15T15:15:31.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.564+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef3c0753a0 con 0x7fef3c0b3160 2024-04-15T15:15:31.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.564+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fef3004b060 con 0x7fef3c0b3160 2024-04-15T15:15:31.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.565+0000 7fef38ff9640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.565+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fef300adc40 con 0x7fef3c0b3160 2024-04-15T15:15:31.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.565+0000 7fef3bfff640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.565+0000 7fef3bfff640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 0x7fef2806edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fef2c001640 tx=0x7fef2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.569+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fef3c0753a0 con 0x7fef3c0b3160 2024-04-15T15:15:31.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.742+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fef3c0689e0 con 0x7fef2806c8f0 2024-04-15T15:15:31.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.750+0000 7fef38ff9640 1 -- 172.21.15.70:0/4159342547 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7fef3c0689e0 con 0x7fef2806c8f0 2024-04-15T15:15:31.755 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:31.757 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 msgr2=0x7fef2806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 0x7fef2806edb0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fef2c001640 tx=0x7fef2c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 msgr2=0x7fef3c153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fef300098c0 tx=0x7fef300046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 shutdown_connections 2024-04-15T15:15:31.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fef3c0d0690 0x7fef3c0b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fef2806c8f0 0x7fef2806edb0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fef3c0b3160 0x7fef3c153740 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 --2- 172.21.15.70:0/4159342547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fef3c0ae280 0x7fef3c15f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.753+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 >> 172.21.15.70:0/4159342547 conn(0x7fef3c06be00 msgr2=0x7fef3c06dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.754+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 shutdown_connections 2024-04-15T15:15:31.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.754+0000 7fef4257c640 1 -- 172.21.15.70:0/4159342547 wait complete. 2024-04-15T15:15:31.766 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-04-15T15:15:30.074062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236128,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738592,"statfs":{"total":773094113280,"available":772852318208,"internally_reserved":0,"allocated":3383296,"data_stored":2027192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002538"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865551+0000","last_change":"2024-04-15T15:15:09.296489+0000","last_active":"2024-04-15T15:15:15.865551+0000","last_peered":"2024-04-15T15:15:15.865551+0000","last_clean":"2024-04-15T15:15:15.865551+0000","last_became_active":"2024-04-15T15:15:09.296225+0000","last_became_peered":"2024-04-15T15:15:09.296225+0000","last_unstale":"2024-04-15T15:15:15.865551+0000","last_undegraded":"2024-04-15T15:15:15.865551+0000","last_fullsized":"2024-04-15T15:15:15.865551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308730+0000","last_change":"2024-04-15T15:15:10.297175+0000","last_active":"2024-04-15T15:15:15.308730+0000","last_peered":"2024-04-15T15:15:15.308730+0000","last_clean":"2024-04-15T15:15:15.308730+0000","last_became_active":"2024-04-15T15:15:10.296707+0000","last_became_peered":"2024-04-15T15:15:10.296707+0000","last_unstale":"2024-04-15T15:15:15.308730+0000","last_undegraded":"2024-04-15T15:15:15.308730+0000","last_fullsized":"2024-04-15T15:15:15.308730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308030+0000","last_change":"2024-04-15T15:15:14.490094+0000","last_active":"2024-04-15T15:15:15.308030+0000","last_peered":"2024-04-15T15:15:15.308030+0000","last_clean":"2024-04-15T15:15:15.308030+0000","last_became_active":"2024-04-15T15:15:14.489727+0000","last_became_peered":"2024-04-15T15:15:14.489727+0000","last_unstale":"2024-04-15T15:15:15.308030+0000","last_undegraded":"2024-04-15T15:15:15.308030+0000","last_fullsized":"2024-04-15T15:15:15.308030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308822+0000","last_change":"2024-04-15T15:15:09.279298+0000","last_active":"2024-04-15T15:15:15.308822+0000","last_peered":"2024-04-15T15:15:15.308822+0000","last_clean":"2024-04-15T15:15:15.308822+0000","last_became_active":"2024-04-15T15:15:09.279018+0000","last_became_peered":"2024-04-15T15:15:09.279018+0000","last_unstale":"2024-04-15T15:15:15.308822+0000","last_undegraded":"2024-04-15T15:15:15.308822+0000","last_fullsized":"2024-04-15T15:15:15.308822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865283+0000","last_change":"2024-04-15T15:15:10.300836+0000","last_active":"2024-04-15T15:15:15.865283+0000","last_peered":"2024-04-15T15:15:15.865283+0000","last_clean":"2024-04-15T15:15:15.865283+0000","last_became_active":"2024-04-15T15:15:10.300574+0000","last_became_peered":"2024-04-15T15:15:10.300574+0000","last_unstale":"2024-04-15T15:15:15.865283+0000","last_undegraded":"2024-04-15T15:15:15.865283+0000","last_fullsized":"2024-04-15T15:15:15.865283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310208+0000","last_change":"2024-04-15T15:15:14.327665+0000","last_active":"2024-04-15T15:15:15.310208+0000","last_peered":"2024-04-15T15:15:15.310208+0000","last_clean":"2024-04-15T15:15:15.310208+0000","last_became_active":"2024-04-15T15:15:14.327161+0000","last_became_peered":"2024-04-15T15:15:14.327161+0000","last_unstale":"2024-04-15T15:15:15.310208+0000","last_undegraded":"2024-04-15T15:15:15.310208+0000","last_fullsized":"2024-04-15T15:15:15.310208+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866087+0000","last_change":"2024-04-15T15:15:09.297431+0000","last_active":"2024-04-15T15:15:15.866087+0000","last_peered":"2024-04-15T15:15:15.866087+0000","last_clean":"2024-04-15T15:15:15.866087+0000","last_became_active":"2024-04-15T15:15:09.297182+0000","last_became_peered":"2024-04-15T15:15:09.297182+0000","last_unstale":"2024-04-15T15:15:15.866087+0000","last_undegraded":"2024-04-15T15:15:15.866087+0000","last_fullsized":"2024-04-15T15:15:15.866087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:45.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865971+0000","last_change":"2024-04-15T15:15:10.303063+0000","last_active":"2024-04-15T15:15:15.865971+0000","last_peered":"2024-04-15T15:15:15.865971+0000","last_clean":"2024-04-15T15:15:15.865971+0000","last_became_active":"2024-04-15T15:15:10.302785+0000","last_became_peered":"2024-04-15T15:15:10.302785+0000","last_unstale":"2024-04-15T15:15:15.865971+0000","last_undegraded":"2024-04-15T15:15:15.865971+0000","last_fullsized":"2024-04-15T15:15:15.865971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309364+0000","last_change":"2024-04-15T15:15:14.328804+0000","last_active":"2024-04-15T15:15:15.309364+0000","last_peered":"2024-04-15T15:15:15.309364+0000","last_clean":"2024-04-15T15:15:15.309364+0000","last_became_active":"2024-04-15T15:15:14.327886+0000","last_became_peered":"2024-04-15T15:15:14.327886+0000","last_unstale":"2024-04-15T15:15:15.309364+0000","last_undegraded":"2024-04-15T15:15:15.309364+0000","last_fullsized":"2024-04-15T15:15:15.309364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:25.064499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865605+0000","last_change":"2024-04-15T15:15:09.300461+0000","last_active":"2024-04-15T15:15:15.865605+0000","last_peered":"2024-04-15T15:15:15.865605+0000","last_clean":"2024-04-15T15:15:15.865605+0000","last_became_active":"2024-04-15T15:15:09.300159+0000","last_became_peered":"2024-04-15T15:15:09.300159+0000","last_unstale":"2024-04-15T15:15:15.865605+0000","last_undegraded":"2024-04-15T15:15:15.865605+0000","last_fullsized":"2024-04-15T15:15:15.865605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865784+0000","last_change":"2024-04-15T15:15:10.298282+0000","last_active":"2024-04-15T15:15:15.865784+0000","last_peered":"2024-04-15T15:15:15.865784+0000","last_clean":"2024-04-15T15:15:15.865784+0000","last_became_active":"2024-04-15T15:15:10.297734+0000","last_became_peered":"2024-04-15T15:15:10.297734+0000","last_unstale":"2024-04-15T15:15:15.865784+0000","last_undegraded":"2024-04-15T15:15:15.865784+0000","last_fullsized":"2024-04-15T15:15:15.865784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309722+0000","last_change":"2024-04-15T15:15:14.321155+0000","last_active":"2024-04-15T15:15:15.309722+0000","last_peered":"2024-04-15T15:15:15.309722+0000","last_clean":"2024-04-15T15:15:15.309722+0000","last_became_active":"2024-04-15T15:15:14.320777+0000","last_became_peered":"2024-04-15T15:15:14.320777+0000","last_unstale":"2024-04-15T15:15:15.309722+0000","last_undegraded":"2024-04-15T15:15:15.309722+0000","last_fullsized":"2024-04-15T15:15:15.309722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:36.915564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310304+0000","last_change":"2024-04-15T15:15:10.288957+0000","last_active":"2024-04-15T15:15:15.310304+0000","last_peered":"2024-04-15T15:15:15.310304+0000","last_clean":"2024-04-15T15:15:15.310304+0000","last_became_active":"2024-04-15T15:15:10.288692+0000","last_became_peered":"2024-04-15T15:15:10.288692+0000","last_unstale":"2024-04-15T15:15:15.310304+0000","last_undegraded":"2024-04-15T15:15:15.310304+0000","last_fullsized":"2024-04-15T15:15:15.310304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865797+0000","last_change":"2024-04-15T15:15:09.295590+0000","last_active":"2024-04-15T15:15:15.865797+0000","last_peered":"2024-04-15T15:15:15.865797+0000","last_clean":"2024-04-15T15:15:15.865797+0000","last_became_active":"2024-04-15T15:15:09.295295+0000","last_became_peered":"2024-04-15T15:15:09.295295+0000","last_unstale":"2024-04-15T15:15:15.865797+0000","last_undegraded":"2024-04-15T15:15:15.865797+0000","last_fullsized":"2024-04-15T15:15:15.865797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309855+0000","last_change":"2024-04-15T15:15:14.327835+0000","last_active":"2024-04-15T15:15:15.309855+0000","last_peered":"2024-04-15T15:15:15.309855+0000","last_clean":"2024-04-15T15:15:15.309855+0000","last_became_active":"2024-04-15T15:15:14.327245+0000","last_became_peered":"2024-04-15T15:15:14.327245+0000","last_unstale":"2024-04-15T15:15:15.309855+0000","last_undegraded":"2024-04-15T15:15:15.309855+0000","last_fullsized":"2024-04-15T15:15:15.309855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308188+0000","last_change":"2024-04-15T15:15:10.295527+0000","last_active":"2024-04-15T15:15:15.308188+0000","last_peered":"2024-04-15T15:15:15.308188+0000","last_clean":"2024-04-15T15:15:15.308188+0000","last_became_active":"2024-04-15T15:15:10.295265+0000","last_became_peered":"2024-04-15T15:15:10.295265+0000","last_unstale":"2024-04-15T15:15:15.308188+0000","last_undegraded":"2024-04-15T15:15:15.308188+0000","last_fullsized":"2024-04-15T15:15:15.308188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393749+0000","last_change":"2024-04-15T15:15:09.291151+0000","last_active":"2024-04-15T15:15:15.393749+0000","last_peered":"2024-04-15T15:15:15.393749+0000","last_clean":"2024-04-15T15:15:15.393749+0000","last_became_active":"2024-04-15T15:15:09.290851+0000","last_became_peered":"2024-04-15T15:15:09.290851+0000","last_unstale":"2024-04-15T15:15:15.393749+0000","last_undegraded":"2024-04-15T15:15:15.393749+0000","last_fullsized":"2024-04-15T15:15:15.393749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309375+0000","last_change":"2024-04-15T15:15:14.327866+0000","last_active":"2024-04-15T15:15:15.309375+0000","last_peered":"2024-04-15T15:15:15.309375+0000","last_clean":"2024-04-15T15:15:15.309375+0000","last_became_active":"2024-04-15T15:15:14.327398+0000","last_became_peered":"2024-04-15T15:15:14.327398+0000","last_unstale":"2024-04-15T15:15:15.309375+0000","last_undegraded":"2024-04-15T15:15:15.309375+0000","last_fullsized":"2024-04-15T15:15:15.309375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:45.919286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308185+0000","last_change":"2024-04-15T15:15:10.294406+0000","last_active":"2024-04-15T15:15:15.308185+0000","last_peered":"2024-04-15T15:15:15.308185+0000","last_clean":"2024-04-15T15:15:15.308185+0000","last_became_active":"2024-04-15T15:15:10.294005+0000","last_became_peered":"2024-04-15T15:15:10.294005+0000","last_unstale":"2024-04-15T15:15:15.308185+0000","last_undegraded":"2024-04-15T15:15:15.308185+0000","last_fullsized":"2024-04-15T15:15:15.308185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393556+0000","last_change":"2024-04-15T15:15:09.288397+0000","last_active":"2024-04-15T15:15:15.393556+0000","last_peered":"2024-04-15T15:15:15.393556+0000","last_clean":"2024-04-15T15:15:15.393556+0000","last_became_active":"2024-04-15T15:15:09.288135+0000","last_became_peered":"2024-04-15T15:15:09.288135+0000","last_unstale":"2024-04-15T15:15:15.393556+0000","last_undegraded":"2024-04-15T15:15:15.393556+0000","last_fullsized":"2024-04-15T15:15:15.393556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394110+0000","last_change":"2024-04-15T15:15:14.488972+0000","last_active":"2024-04-15T15:15:15.394110+0000","last_peered":"2024-04-15T15:15:15.394110+0000","last_clean":"2024-04-15T15:15:15.394110+0000","last_became_active":"2024-04-15T15:15:14.488302+0000","last_became_peered":"2024-04-15T15:15:14.488302+0000","last_unstale":"2024-04-15T15:15:15.394110+0000","last_undegraded":"2024-04-15T15:15:15.394110+0000","last_fullsized":"2024-04-15T15:15:15.394110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310517+0000","last_change":"2024-04-15T15:15:10.294428+0000","last_active":"2024-04-15T15:15:15.310517+0000","last_peered":"2024-04-15T15:15:15.310517+0000","last_clean":"2024-04-15T15:15:15.310517+0000","last_became_active":"2024-04-15T15:15:10.294121+0000","last_became_peered":"2024-04-15T15:15:10.294121+0000","last_unstale":"2024-04-15T15:15:15.310517+0000","last_undegraded":"2024-04-15T15:15:15.310517+0000","last_fullsized":"2024-04-15T15:15:15.310517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:06.364679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865099+0000","last_change":"2024-04-15T15:15:09.294900+0000","last_active":"2024-04-15T15:15:15.865099+0000","last_peered":"2024-04-15T15:15:15.865099+0000","last_clean":"2024-04-15T15:15:15.865099+0000","last_became_active":"2024-04-15T15:15:09.294601+0000","last_became_peered":"2024-04-15T15:15:09.294601+0000","last_unstale":"2024-04-15T15:15:15.865099+0000","last_undegraded":"2024-04-15T15:15:15.865099+0000","last_fullsized":"2024-04-15T15:15:15.865099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309437+0000","last_change":"2024-04-15T15:15:14.328664+0000","last_active":"2024-04-15T15:15:15.309437+0000","last_peered":"2024-04-15T15:15:15.309437+0000","last_clean":"2024-04-15T15:15:15.309437+0000","last_became_active":"2024-04-15T15:15:14.327796+0000","last_became_peered":"2024-04-15T15:15:14.327796+0000","last_unstale":"2024-04-15T15:15:15.309437+0000","last_undegraded":"2024-04-15T15:15:15.309437+0000","last_fullsized":"2024-04-15T15:15:15.309437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:28.892025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865447+0000","last_change":"2024-04-15T15:15:10.288747+0000","last_active":"2024-04-15T15:15:15.865447+0000","last_peered":"2024-04-15T15:15:15.865447+0000","last_clean":"2024-04-15T15:15:15.865447+0000","last_became_active":"2024-04-15T15:15:10.288465+0000","last_became_peered":"2024-04-15T15:15:10.288465+0000","last_unstale":"2024-04-15T15:15:15.865447+0000","last_undegraded":"2024-04-15T15:15:15.865447+0000","last_fullsized":"2024-04-15T15:15:15.865447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311197+0000","last_change":"2024-04-15T15:15:09.291999+0000","last_active":"2024-04-15T15:15:15.311197+0000","last_peered":"2024-04-15T15:15:15.311197+0000","last_clean":"2024-04-15T15:15:15.311197+0000","last_became_active":"2024-04-15T15:15:09.291744+0000","last_became_peered":"2024-04-15T15:15:09.291744+0000","last_unstale":"2024-04-15T15:15:15.311197+0000","last_undegraded":"2024-04-15T15:15:15.311197+0000","last_fullsized":"2024-04-15T15:15:15.311197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310618+0000","last_change":"2024-04-15T15:15:14.491129+0000","last_active":"2024-04-15T15:15:15.310618+0000","last_peered":"2024-04-15T15:15:15.310618+0000","last_clean":"2024-04-15T15:15:15.310618+0000","last_became_active":"2024-04-15T15:15:14.490619+0000","last_became_peered":"2024-04-15T15:15:14.490619+0000","last_unstale":"2024-04-15T15:15:15.310618+0000","last_undegraded":"2024-04-15T15:15:15.310618+0000","last_fullsized":"2024-04-15T15:15:15.310618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:46.083837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308841+0000","last_change":"2024-04-15T15:15:10.297516+0000","last_active":"2024-04-15T15:15:15.308841+0000","last_peered":"2024-04-15T15:15:15.308841+0000","last_clean":"2024-04-15T15:15:15.308841+0000","last_became_active":"2024-04-15T15:15:10.297282+0000","last_became_peered":"2024-04-15T15:15:10.297282+0000","last_unstale":"2024-04-15T15:15:15.308841+0000","last_undegraded":"2024-04-15T15:15:15.308841+0000","last_fullsized":"2024-04-15T15:15:15.308841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394314+0000","last_change":"2024-04-15T15:15:09.288389+0000","last_active":"2024-04-15T15:15:15.394314+0000","last_peered":"2024-04-15T15:15:15.394314+0000","last_clean":"2024-04-15T15:15:15.394314+0000","last_became_active":"2024-04-15T15:15:09.288101+0000","last_became_peered":"2024-04-15T15:15:09.288101+0000","last_unstale":"2024-04-15T15:15:15.394314+0000","last_undegraded":"2024-04-15T15:15:15.394314+0000","last_fullsized":"2024-04-15T15:15:15.394314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.307926+0000","last_change":"2024-04-15T15:15:14.488661+0000","last_active":"2024-04-15T15:15:15.307926+0000","last_peered":"2024-04-15T15:15:15.307926+0000","last_clean":"2024-04-15T15:15:15.307926+0000","last_became_active":"2024-04-15T15:15:14.487944+0000","last_became_peered":"2024-04-15T15:15:14.487944+0000","last_unstale":"2024-04-15T15:15:15.307926+0000","last_undegraded":"2024-04-15T15:15:15.307926+0000","last_fullsized":"2024-04-15T15:15:15.307926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:36.475872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393929+0000","last_change":"2024-04-15T15:15:10.289924+0000","last_active":"2024-04-15T15:15:15.393929+0000","last_peered":"2024-04-15T15:15:15.393929+0000","last_clean":"2024-04-15T15:15:15.393929+0000","last_became_active":"2024-04-15T15:15:10.289662+0000","last_became_peered":"2024-04-15T15:15:10.289662+0000","last_unstale":"2024-04-15T15:15:15.393929+0000","last_undegraded":"2024-04-15T15:15:15.393929+0000","last_fullsized":"2024-04-15T15:15:15.393929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866240+0000","last_change":"2024-04-15T15:15:09.295866+0000","last_active":"2024-04-15T15:15:15.866240+0000","last_peered":"2024-04-15T15:15:15.866240+0000","last_clean":"2024-04-15T15:15:15.866240+0000","last_became_active":"2024-04-15T15:15:09.295507+0000","last_became_peered":"2024-04-15T15:15:09.295507+0000","last_unstale":"2024-04-15T15:15:15.866240+0000","last_undegraded":"2024-04-15T15:15:15.866240+0000","last_fullsized":"2024-04-15T15:15:15.866240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:01.918524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865520+0000","last_change":"2024-04-15T15:15:14.487233+0000","last_active":"2024-04-15T15:15:15.865520+0000","last_peered":"2024-04-15T15:15:15.865520+0000","last_clean":"2024-04-15T15:15:15.865520+0000","last_became_active":"2024-04-15T15:15:14.486846+0000","last_became_peered":"2024-04-15T15:15:14.486846+0000","last_unstale":"2024-04-15T15:15:15.865520+0000","last_undegraded":"2024-04-15T15:15:15.865520+0000","last_fullsized":"2024-04-15T15:15:15.865520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865899+0000","last_change":"2024-04-15T15:15:10.299154+0000","last_active":"2024-04-15T15:15:15.865899+0000","last_peered":"2024-04-15T15:15:15.865899+0000","last_clean":"2024-04-15T15:15:15.865899+0000","last_became_active":"2024-04-15T15:15:10.298711+0000","last_became_peered":"2024-04-15T15:15:10.298711+0000","last_unstale":"2024-04-15T15:15:15.865899+0000","last_undegraded":"2024-04-15T15:15:15.865899+0000","last_fullsized":"2024-04-15T15:15:15.865899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394042+0000","last_change":"2024-04-15T15:15:09.289347+0000","last_active":"2024-04-15T15:15:15.394042+0000","last_peered":"2024-04-15T15:15:15.394042+0000","last_clean":"2024-04-15T15:15:15.394042+0000","last_became_active":"2024-04-15T15:15:09.289062+0000","last_became_peered":"2024-04-15T15:15:09.289062+0000","last_unstale":"2024-04-15T15:15:15.394042+0000","last_undegraded":"2024-04-15T15:15:15.394042+0000","last_fullsized":"2024-04-15T15:15:15.394042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393588+0000","last_change":"2024-04-15T15:15:14.330539+0000","last_active":"2024-04-15T15:15:15.393588+0000","last_peered":"2024-04-15T15:15:15.393588+0000","last_clean":"2024-04-15T15:15:15.393588+0000","last_became_active":"2024-04-15T15:15:14.330173+0000","last_became_peered":"2024-04-15T15:15:14.330173+0000","last_unstale":"2024-04-15T15:15:15.393588+0000","last_undegraded":"2024-04-15T15:15:15.393588+0000","last_fullsized":"2024-04-15T15:15:15.393588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310032+0000","last_change":"2024-04-15T15:15:10.286603+0000","last_active":"2024-04-15T15:15:15.310032+0000","last_peered":"2024-04-15T15:15:15.310032+0000","last_clean":"2024-04-15T15:15:15.310032+0000","last_became_active":"2024-04-15T15:15:10.286306+0000","last_became_peered":"2024-04-15T15:15:10.286306+0000","last_unstale":"2024-04-15T15:15:15.310032+0000","last_undegraded":"2024-04-15T15:15:15.310032+0000","last_fullsized":"2024-04-15T15:15:15.310032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:47.785526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865601+0000","last_change":"2024-04-15T15:15:09.291918+0000","last_active":"2024-04-15T15:15:15.865601+0000","last_peered":"2024-04-15T15:15:15.865601+0000","last_clean":"2024-04-15T15:15:15.865601+0000","last_became_active":"2024-04-15T15:15:09.291664+0000","last_became_peered":"2024-04-15T15:15:09.291664+0000","last_unstale":"2024-04-15T15:15:15.865601+0000","last_undegraded":"2024-04-15T15:15:15.865601+0000","last_fullsized":"2024-04-15T15:15:15.865601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393983+0000","last_change":"2024-04-15T15:15:14.329421+0000","last_active":"2024-04-15T15:15:15.393983+0000","last_peered":"2024-04-15T15:15:15.393983+0000","last_clean":"2024-04-15T15:15:15.393983+0000","last_became_active":"2024-04-15T15:15:14.329120+0000","last_became_peered":"2024-04-15T15:15:14.329120+0000","last_unstale":"2024-04-15T15:15:15.393983+0000","last_undegraded":"2024-04-15T15:15:15.393983+0000","last_fullsized":"2024-04-15T15:15:15.393983+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393608+0000","last_change":"2024-04-15T15:15:10.290334+0000","last_active":"2024-04-15T15:15:15.393608+0000","last_peered":"2024-04-15T15:15:15.393608+0000","last_clean":"2024-04-15T15:15:15.393608+0000","last_became_active":"2024-04-15T15:15:10.290075+0000","last_became_peered":"2024-04-15T15:15:10.290075+0000","last_unstale":"2024-04-15T15:15:15.393608+0000","last_undegraded":"2024-04-15T15:15:15.393608+0000","last_fullsized":"2024-04-15T15:15:15.393608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864908+0000","last_change":"2024-04-15T15:15:09.291564+0000","last_active":"2024-04-15T15:15:15.864908+0000","last_peered":"2024-04-15T15:15:15.864908+0000","last_clean":"2024-04-15T15:15:15.864908+0000","last_became_active":"2024-04-15T15:15:09.291265+0000","last_became_peered":"2024-04-15T15:15:09.291265+0000","last_unstale":"2024-04-15T15:15:15.864908+0000","last_undegraded":"2024-04-15T15:15:15.864908+0000","last_fullsized":"2024-04-15T15:15:15.864908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394137+0000","last_change":"2024-04-15T15:15:14.329535+0000","last_active":"2024-04-15T15:15:15.394137+0000","last_peered":"2024-04-15T15:15:15.394137+0000","last_clean":"2024-04-15T15:15:15.394137+0000","last_became_active":"2024-04-15T15:15:14.329173+0000","last_became_peered":"2024-04-15T15:15:14.329173+0000","last_unstale":"2024-04-15T15:15:15.394137+0000","last_undegraded":"2024-04-15T15:15:15.394137+0000","last_fullsized":"2024-04-15T15:15:15.394137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864715+0000","last_change":"2024-04-15T15:15:10.292048+0000","last_active":"2024-04-15T15:15:15.864715+0000","last_peered":"2024-04-15T15:15:15.864715+0000","last_clean":"2024-04-15T15:15:15.864715+0000","last_became_active":"2024-04-15T15:15:10.291773+0000","last_became_peered":"2024-04-15T15:15:10.291773+0000","last_unstale":"2024-04-15T15:15:15.864715+0000","last_undegraded":"2024-04-15T15:15:15.864715+0000","last_fullsized":"2024-04-15T15:15:15.864715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:45.215354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865214+0000","last_change":"2024-04-15T15:15:09.295426+0000","last_active":"2024-04-15T15:15:15.865214+0000","last_peered":"2024-04-15T15:15:15.865214+0000","last_clean":"2024-04-15T15:15:15.865214+0000","last_became_active":"2024-04-15T15:15:09.295096+0000","last_became_peered":"2024-04-15T15:15:09.295096+0000","last_unstale":"2024-04-15T15:15:15.865214+0000","last_undegraded":"2024-04-15T15:15:15.865214+0000","last_fullsized":"2024-04-15T15:15:15.865214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865346+0000","last_change":"2024-04-15T15:15:14.487282+0000","last_active":"2024-04-15T15:15:15.865346+0000","last_peered":"2024-04-15T15:15:15.865346+0000","last_clean":"2024-04-15T15:15:15.865346+0000","last_became_active":"2024-04-15T15:15:14.486873+0000","last_became_peered":"2024-04-15T15:15:14.486873+0000","last_unstale":"2024-04-15T15:15:15.865346+0000","last_undegraded":"2024-04-15T15:15:15.865346+0000","last_fullsized":"2024-04-15T15:15:15.865346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308149+0000","last_change":"2024-04-15T15:15:10.301516+0000","last_active":"2024-04-15T15:15:15.308149+0000","last_peered":"2024-04-15T15:15:15.308149+0000","last_clean":"2024-04-15T15:15:15.308149+0000","last_became_active":"2024-04-15T15:15:10.301272+0000","last_became_peered":"2024-04-15T15:15:10.301272+0000","last_unstale":"2024-04-15T15:15:15.308149+0000","last_undegraded":"2024-04-15T15:15:15.308149+0000","last_fullsized":"2024-04-15T15:15:15.308149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865920+0000","last_change":"2024-04-15T15:15:09.292652+0000","last_active":"2024-04-15T15:15:15.865920+0000","last_peered":"2024-04-15T15:15:15.865920+0000","last_clean":"2024-04-15T15:15:15.865920+0000","last_became_active":"2024-04-15T15:15:09.292366+0000","last_became_peered":"2024-04-15T15:15:09.292366+0000","last_unstale":"2024-04-15T15:15:15.865920+0000","last_undegraded":"2024-04-15T15:15:15.865920+0000","last_fullsized":"2024-04-15T15:15:15.865920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865328+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:15:15.865328+0000","last_peered":"2024-04-15T15:15:15.865328+0000","last_clean":"2024-04-15T15:15:15.865328+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:15:15.865328+0000","last_undegraded":"2024-04-15T15:15:15.865328+0000","last_fullsized":"2024-04-15T15:15:15.865328+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310218+0000","last_change":"2024-04-15T15:15:14.488973+0000","last_active":"2024-04-15T15:15:15.310218+0000","last_peered":"2024-04-15T15:15:15.310218+0000","last_clean":"2024-04-15T15:15:15.310218+0000","last_became_active":"2024-04-15T15:15:14.488151+0000","last_became_peered":"2024-04-15T15:15:14.488151+0000","last_unstale":"2024-04-15T15:15:15.310218+0000","last_undegraded":"2024-04-15T15:15:15.310218+0000","last_fullsized":"2024-04-15T15:15:15.310218+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865992+0000","last_change":"2024-04-15T15:15:10.303487+0000","last_active":"2024-04-15T15:15:15.865992+0000","last_peered":"2024-04-15T15:15:15.865992+0000","last_clean":"2024-04-15T15:15:15.865992+0000","last_became_active":"2024-04-15T15:15:10.303222+0000","last_became_peered":"2024-04-15T15:15:10.303222+0000","last_unstale":"2024-04-15T15:15:15.865992+0000","last_undegraded":"2024-04-15T15:15:15.865992+0000","last_fullsized":"2024-04-15T15:15:15.865992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:09.292763+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:09.292482+0000","last_became_peered":"2024-04-15T15:15:09.292482+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310239+0000","last_change":"2024-04-15T15:15:14.487139+0000","last_active":"2024-04-15T15:15:15.310239+0000","last_peered":"2024-04-15T15:15:15.310239+0000","last_clean":"2024-04-15T15:15:15.310239+0000","last_became_active":"2024-04-15T15:15:14.486746+0000","last_became_peered":"2024-04-15T15:15:14.486746+0000","last_unstale":"2024-04-15T15:15:15.310239+0000","last_undegraded":"2024-04-15T15:15:15.310239+0000","last_fullsized":"2024-04-15T15:15:15.310239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865734+0000","last_change":"2024-04-15T15:15:10.298528+0000","last_active":"2024-04-15T15:15:15.865734+0000","last_peered":"2024-04-15T15:15:15.865734+0000","last_clean":"2024-04-15T15:15:15.865734+0000","last_became_active":"2024-04-15T15:15:10.298175+0000","last_became_peered":"2024-04-15T15:15:10.298175+0000","last_unstale":"2024-04-15T15:15:15.865734+0000","last_undegraded":"2024-04-15T15:15:15.865734+0000","last_fullsized":"2024-04-15T15:15:15.865734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394078+0000","last_change":"2024-04-15T15:15:09.287274+0000","last_active":"2024-04-15T15:15:15.394078+0000","last_peered":"2024-04-15T15:15:15.394078+0000","last_clean":"2024-04-15T15:15:15.394078+0000","last_became_active":"2024-04-15T15:15:09.287022+0000","last_became_peered":"2024-04-15T15:15:09.287022+0000","last_unstale":"2024-04-15T15:15:15.394078+0000","last_undegraded":"2024-04-15T15:15:15.394078+0000","last_fullsized":"2024-04-15T15:15:15.394078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865617+0000","last_change":"2024-04-15T15:15:14.327367+0000","last_active":"2024-04-15T15:15:15.865617+0000","last_peered":"2024-04-15T15:15:15.865617+0000","last_clean":"2024-04-15T15:15:15.865617+0000","last_became_active":"2024-04-15T15:15:14.327012+0000","last_became_peered":"2024-04-15T15:15:14.327012+0000","last_unstale":"2024-04-15T15:15:15.865617+0000","last_undegraded":"2024-04-15T15:15:15.865617+0000","last_fullsized":"2024-04-15T15:15:15.865617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866362+0000","last_change":"2024-04-15T15:15:10.300664+0000","last_active":"2024-04-15T15:15:15.866362+0000","last_peered":"2024-04-15T15:15:15.866362+0000","last_clean":"2024-04-15T15:15:15.866362+0000","last_became_active":"2024-04-15T15:15:10.300411+0000","last_became_peered":"2024-04-15T15:15:10.300411+0000","last_unstale":"2024-04-15T15:15:15.866362+0000","last_undegraded":"2024-04-15T15:15:15.866362+0000","last_fullsized":"2024-04-15T15:15:15.866362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:32.450402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864760+0000","last_change":"2024-04-15T15:15:09.291588+0000","last_active":"2024-04-15T15:15:15.864760+0000","last_peered":"2024-04-15T15:15:15.864760+0000","last_clean":"2024-04-15T15:15:15.864760+0000","last_became_active":"2024-04-15T15:15:09.291317+0000","last_became_peered":"2024-04-15T15:15:09.291317+0000","last_unstale":"2024-04-15T15:15:15.864760+0000","last_undegraded":"2024-04-15T15:15:15.864760+0000","last_fullsized":"2024-04-15T15:15:15.864760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:14.488073+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:14.487630+0000","last_became_peered":"2024-04-15T15:15:14.487630+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866072+0000","last_change":"2024-04-15T15:15:10.293789+0000","last_active":"2024-04-15T15:15:15.866072+0000","last_peered":"2024-04-15T15:15:15.866072+0000","last_clean":"2024-04-15T15:15:15.866072+0000","last_became_active":"2024-04-15T15:15:10.293508+0000","last_became_peered":"2024-04-15T15:15:10.293508+0000","last_unstale":"2024-04-15T15:15:15.866072+0000","last_undegraded":"2024-04-15T15:15:15.866072+0000","last_fullsized":"2024-04-15T15:15:15.866072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:13.731730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310273+0000","last_change":"2024-04-15T15:15:09.280627+0000","last_active":"2024-04-15T15:15:15.310273+0000","last_peered":"2024-04-15T15:15:15.310273+0000","last_clean":"2024-04-15T15:15:15.310273+0000","last_became_active":"2024-04-15T15:15:09.280327+0000","last_became_peered":"2024-04-15T15:15:09.280327+0000","last_unstale":"2024-04-15T15:15:15.310273+0000","last_undegraded":"2024-04-15T15:15:15.310273+0000","last_fullsized":"2024-04-15T15:15:15.310273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:24.741048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308985+0000","last_change":"2024-04-15T15:15:14.327503+0000","last_active":"2024-04-15T15:15:15.308985+0000","last_peered":"2024-04-15T15:15:15.308985+0000","last_clean":"2024-04-15T15:15:15.308985+0000","last_became_active":"2024-04-15T15:15:14.327045+0000","last_became_peered":"2024-04-15T15:15:14.327045+0000","last_unstale":"2024-04-15T15:15:15.308985+0000","last_undegraded":"2024-04-15T15:15:15.308985+0000","last_fullsized":"2024-04-15T15:15:15.308985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:41.707546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865253+0000","last_change":"2024-04-15T15:15:10.294511+0000","last_active":"2024-04-15T15:15:15.865253+0000","last_peered":"2024-04-15T15:15:15.865253+0000","last_clean":"2024-04-15T15:15:15.865253+0000","last_became_active":"2024-04-15T15:15:10.294217+0000","last_became_peered":"2024-04-15T15:15:10.294217+0000","last_unstale":"2024-04-15T15:15:15.865253+0000","last_undegraded":"2024-04-15T15:15:15.865253+0000","last_fullsized":"2024-04-15T15:15:15.865253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864350+0000","last_change":"2024-04-15T15:15:09.289902+0000","last_active":"2024-04-15T15:15:15.864350+0000","last_peered":"2024-04-15T15:15:15.864350+0000","last_clean":"2024-04-15T15:15:15.864350+0000","last_became_active":"2024-04-15T15:15:09.289521+0000","last_became_peered":"2024-04-15T15:15:09.289521+0000","last_unstale":"2024-04-15T15:15:15.864350+0000","last_undegraded":"2024-04-15T15:15:15.864350+0000","last_fullsized":"2024-04-15T15:15:15.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865201+0000","last_change":"2024-04-15T15:15:14.325169+0000","last_active":"2024-04-15T15:15:15.865201+0000","last_peered":"2024-04-15T15:15:15.865201+0000","last_clean":"2024-04-15T15:15:15.865201+0000","last_became_active":"2024-04-15T15:15:14.324680+0000","last_became_peered":"2024-04-15T15:15:14.324680+0000","last_unstale":"2024-04-15T15:15:15.865201+0000","last_undegraded":"2024-04-15T15:15:15.865201+0000","last_fullsized":"2024-04-15T15:15:15.865201+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308937+0000","last_change":"2024-04-15T15:15:10.291452+0000","last_active":"2024-04-15T15:15:15.308937+0000","last_peered":"2024-04-15T15:15:15.308937+0000","last_clean":"2024-04-15T15:15:15.308937+0000","last_became_active":"2024-04-15T15:15:10.291170+0000","last_became_peered":"2024-04-15T15:15:10.291170+0000","last_unstale":"2024-04-15T15:15:15.308937+0000","last_undegraded":"2024-04-15T15:15:15.308937+0000","last_fullsized":"2024-04-15T15:15:15.308937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309951+0000","last_change":"2024-04-15T15:15:09.292208+0000","last_active":"2024-04-15T15:15:15.309951+0000","last_peered":"2024-04-15T15:15:15.309951+0000","last_clean":"2024-04-15T15:15:15.309951+0000","last_became_active":"2024-04-15T15:15:09.290702+0000","last_became_peered":"2024-04-15T15:15:09.290702+0000","last_unstale":"2024-04-15T15:15:15.309951+0000","last_undegraded":"2024-04-15T15:15:15.309951+0000","last_fullsized":"2024-04-15T15:15:15.309951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308793+0000","last_change":"2024-04-15T15:15:14.488657+0000","last_active":"2024-04-15T15:15:15.308793+0000","last_peered":"2024-04-15T15:15:15.308793+0000","last_clean":"2024-04-15T15:15:15.308793+0000","last_became_active":"2024-04-15T15:15:14.488282+0000","last_became_peered":"2024-04-15T15:15:14.488282+0000","last_unstale":"2024-04-15T15:15:15.308793+0000","last_undegraded":"2024-04-15T15:15:15.308793+0000","last_fullsized":"2024-04-15T15:15:15.308793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:21:03.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865696+0000","last_change":"2024-04-15T15:15:10.301823+0000","last_active":"2024-04-15T15:15:15.865696+0000","last_peered":"2024-04-15T15:15:15.865696+0000","last_clean":"2024-04-15T15:15:15.865696+0000","last_became_active":"2024-04-15T15:15:10.301545+0000","last_became_peered":"2024-04-15T15:15:10.301545+0000","last_unstale":"2024-04-15T15:15:15.865696+0000","last_undegraded":"2024-04-15T15:15:15.865696+0000","last_fullsized":"2024-04-15T15:15:15.865696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865481+0000","last_change":"2024-04-15T15:15:09.291724+0000","last_active":"2024-04-15T15:15:15.865481+0000","last_peered":"2024-04-15T15:15:15.865481+0000","last_clean":"2024-04-15T15:15:15.865481+0000","last_became_active":"2024-04-15T15:15:09.291390+0000","last_became_peered":"2024-04-15T15:15:09.291390+0000","last_unstale":"2024-04-15T15:15:15.865481+0000","last_undegraded":"2024-04-15T15:15:15.865481+0000","last_fullsized":"2024-04-15T15:15:15.865481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308524+0000","last_change":"2024-04-15T15:15:14.491287+0000","last_active":"2024-04-15T15:15:15.308524+0000","last_peered":"2024-04-15T15:15:15.308524+0000","last_clean":"2024-04-15T15:15:15.308524+0000","last_became_active":"2024-04-15T15:15:14.488643+0000","last_became_peered":"2024-04-15T15:15:14.488643+0000","last_unstale":"2024-04-15T15:15:15.308524+0000","last_undegraded":"2024-04-15T15:15:15.308524+0000","last_fullsized":"2024-04-15T15:15:15.308524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:29:47.252030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865184+0000","last_change":"2024-04-15T15:15:10.293687+0000","last_active":"2024-04-15T15:15:15.865184+0000","last_peered":"2024-04-15T15:15:15.865184+0000","last_clean":"2024-04-15T15:15:15.865184+0000","last_became_active":"2024-04-15T15:15:10.293430+0000","last_became_peered":"2024-04-15T15:15:10.293430+0000","last_unstale":"2024-04-15T15:15:15.865184+0000","last_undegraded":"2024-04-15T15:15:15.865184+0000","last_fullsized":"2024-04-15T15:15:15.865184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865653+0000","last_change":"2024-04-15T15:15:09.282027+0000","last_active":"2024-04-15T15:15:15.865653+0000","last_peered":"2024-04-15T15:15:15.865653+0000","last_clean":"2024-04-15T15:15:15.865653+0000","last_became_active":"2024-04-15T15:15:09.281730+0000","last_became_peered":"2024-04-15T15:15:09.281730+0000","last_unstale":"2024-04-15T15:15:15.865653+0000","last_undegraded":"2024-04-15T15:15:15.865653+0000","last_fullsized":"2024-04-15T15:15:15.865653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:01.954710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866059+0000","last_change":"2024-04-15T15:15:14.489189+0000","last_active":"2024-04-15T15:15:15.866059+0000","last_peered":"2024-04-15T15:15:15.866059+0000","last_clean":"2024-04-15T15:15:15.866059+0000","last_became_active":"2024-04-15T15:15:14.488758+0000","last_became_peered":"2024-04-15T15:15:14.488758+0000","last_unstale":"2024-04-15T15:15:15.866059+0000","last_undegraded":"2024-04-15T15:15:15.866059+0000","last_fullsized":"2024-04-15T15:15:15.866059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865898+0000","last_change":"2024-04-15T15:15:10.298047+0000","last_active":"2024-04-15T15:15:15.865898+0000","last_peered":"2024-04-15T15:15:15.865898+0000","last_clean":"2024-04-15T15:15:15.865898+0000","last_became_active":"2024-04-15T15:15:10.297779+0000","last_became_peered":"2024-04-15T15:15:10.297779+0000","last_unstale":"2024-04-15T15:15:15.865898+0000","last_undegraded":"2024-04-15T15:15:15.865898+0000","last_fullsized":"2024-04-15T15:15:15.865898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:19.789685+0000","last_change":"2024-04-15T15:15:09.290413+0000","last_active":"2024-04-15T15:15:19.789685+0000","last_peered":"2024-04-15T15:15:19.789685+0000","last_clean":"2024-04-15T15:15:19.789685+0000","last_became_active":"2024-04-15T15:15:09.289971+0000","last_became_peered":"2024-04-15T15:15:09.289971+0000","last_unstale":"2024-04-15T15:15:19.789685+0000","last_undegraded":"2024-04-15T15:15:19.789685+0000","last_fullsized":"2024-04-15T15:15:19.789685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393701+0000","last_change":"2024-04-15T15:15:14.324351+0000","last_active":"2024-04-15T15:15:15.393701+0000","last_peered":"2024-04-15T15:15:15.393701+0000","last_clean":"2024-04-15T15:15:15.393701+0000","last_became_active":"2024-04-15T15:15:14.322177+0000","last_became_peered":"2024-04-15T15:15:14.322177+0000","last_unstale":"2024-04-15T15:15:15.393701+0000","last_undegraded":"2024-04-15T15:15:15.393701+0000","last_fullsized":"2024-04-15T15:15:15.393701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310092+0000","last_change":"2024-04-15T15:15:10.290888+0000","last_active":"2024-04-15T15:15:15.310092+0000","last_peered":"2024-04-15T15:15:15.310092+0000","last_clean":"2024-04-15T15:15:15.310092+0000","last_became_active":"2024-04-15T15:15:10.290630+0000","last_became_peered":"2024-04-15T15:15:10.290630+0000","last_unstale":"2024-04-15T15:15:15.310092+0000","last_undegraded":"2024-04-15T15:15:15.310092+0000","last_fullsized":"2024-04-15T15:15:15.310092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394037+0000","last_change":"2024-04-15T15:15:09.291950+0000","last_active":"2024-04-15T15:15:15.394037+0000","last_peered":"2024-04-15T15:15:15.394037+0000","last_clean":"2024-04-15T15:15:15.394037+0000","last_became_active":"2024-04-15T15:15:09.291687+0000","last_became_peered":"2024-04-15T15:15:09.291687+0000","last_unstale":"2024-04-15T15:15:15.394037+0000","last_undegraded":"2024-04-15T15:15:15.394037+0000","last_fullsized":"2024-04-15T15:15:15.394037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309744+0000","last_change":"2024-04-15T15:15:14.328229+0000","last_active":"2024-04-15T15:15:15.309744+0000","last_peered":"2024-04-15T15:15:15.309744+0000","last_clean":"2024-04-15T15:15:15.309744+0000","last_became_active":"2024-04-15T15:15:14.327167+0000","last_became_peered":"2024-04-15T15:15:14.327167+0000","last_unstale":"2024-04-15T15:15:15.309744+0000","last_undegraded":"2024-04-15T15:15:15.309744+0000","last_fullsized":"2024-04-15T15:15:15.309744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:36.054042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865433+0000","last_change":"2024-04-15T15:15:10.300821+0000","last_active":"2024-04-15T15:15:15.865433+0000","last_peered":"2024-04-15T15:15:15.865433+0000","last_clean":"2024-04-15T15:15:15.865433+0000","last_became_active":"2024-04-15T15:15:10.300561+0000","last_became_peered":"2024-04-15T15:15:10.300561+0000","last_unstale":"2024-04-15T15:15:15.865433+0000","last_undegraded":"2024-04-15T15:15:15.865433+0000","last_fullsized":"2024-04-15T15:15:15.865433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865315+0000","last_change":"2024-04-15T15:15:09.294709+0000","last_active":"2024-04-15T15:15:15.865315+0000","last_peered":"2024-04-15T15:15:15.865315+0000","last_clean":"2024-04-15T15:15:15.865315+0000","last_became_active":"2024-04-15T15:15:09.294135+0000","last_became_peered":"2024-04-15T15:15:09.294135+0000","last_unstale":"2024-04-15T15:15:15.865315+0000","last_undegraded":"2024-04-15T15:15:15.865315+0000","last_fullsized":"2024-04-15T15:15:15.865315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308127+0000","last_change":"2024-04-15T15:15:14.332302+0000","last_active":"2024-04-15T15:15:15.308127+0000","last_peered":"2024-04-15T15:15:15.308127+0000","last_clean":"2024-04-15T15:15:15.308127+0000","last_became_active":"2024-04-15T15:15:14.331893+0000","last_became_peered":"2024-04-15T15:15:14.331893+0000","last_unstale":"2024-04-15T15:15:15.308127+0000","last_undegraded":"2024-04-15T15:15:15.308127+0000","last_fullsized":"2024-04-15T15:15:15.308127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:18:57.970154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865270+0000","last_change":"2024-04-15T15:15:10.295887+0000","last_active":"2024-04-15T15:15:15.865270+0000","last_peered":"2024-04-15T15:15:15.865270+0000","last_clean":"2024-04-15T15:15:15.865270+0000","last_became_active":"2024-04-15T15:15:10.295591+0000","last_became_peered":"2024-04-15T15:15:10.295591+0000","last_unstale":"2024-04-15T15:15:15.865270+0000","last_undegraded":"2024-04-15T15:15:15.865270+0000","last_fullsized":"2024-04-15T15:15:15.865270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311117+0000","last_change":"2024-04-15T15:15:09.290489+0000","last_active":"2024-04-15T15:15:15.311117+0000","last_peered":"2024-04-15T15:15:15.311117+0000","last_clean":"2024-04-15T15:15:15.311117+0000","last_became_active":"2024-04-15T15:15:09.290207+0000","last_became_peered":"2024-04-15T15:15:09.290207+0000","last_unstale":"2024-04-15T15:15:15.311117+0000","last_undegraded":"2024-04-15T15:15:15.311117+0000","last_fullsized":"2024-04-15T15:15:15.311117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308129+0000","last_change":"2024-04-15T15:15:14.490596+0000","last_active":"2024-04-15T15:15:15.308129+0000","last_peered":"2024-04-15T15:15:15.308129+0000","last_clean":"2024-04-15T15:15:15.308129+0000","last_became_active":"2024-04-15T15:15:14.489609+0000","last_became_peered":"2024-04-15T15:15:14.489609+0000","last_unstale":"2024-04-15T15:15:15.308129+0000","last_undegraded":"2024-04-15T15:15:15.308129+0000","last_fullsized":"2024-04-15T15:15:15.308129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865063+0000","last_change":"2024-04-15T15:15:14.489631+0000","last_active":"2024-04-15T15:15:15.865063+0000","last_peered":"2024-04-15T15:15:15.865063+0000","last_clean":"2024-04-15T15:15:15.865063+0000","last_became_active":"2024-04-15T15:15:14.489248+0000","last_became_peered":"2024-04-15T15:15:14.489248+0000","last_unstale":"2024-04-15T15:15:15.865063+0000","last_undegraded":"2024-04-15T15:15:15.865063+0000","last_fullsized":"2024-04-15T15:15:15.865063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393445+0000","last_change":"2024-04-15T15:15:10.293423+0000","last_active":"2024-04-15T15:15:15.393445+0000","last_peered":"2024-04-15T15:15:15.393445+0000","last_clean":"2024-04-15T15:15:15.393445+0000","last_became_active":"2024-04-15T15:15:10.293101+0000","last_became_peered":"2024-04-15T15:15:10.293101+0000","last_unstale":"2024-04-15T15:15:15.393445+0000","last_undegraded":"2024-04-15T15:15:15.393445+0000","last_fullsized":"2024-04-15T15:15:15.393445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865159+0000","last_change":"2024-04-15T15:15:09.290248+0000","last_active":"2024-04-15T15:15:15.865159+0000","last_peered":"2024-04-15T15:15:15.865159+0000","last_clean":"2024-04-15T15:15:15.865159+0000","last_became_active":"2024-04-15T15:15:09.289792+0000","last_became_peered":"2024-04-15T15:15:09.289792+0000","last_unstale":"2024-04-15T15:15:15.865159+0000","last_undegraded":"2024-04-15T15:15:15.865159+0000","last_fullsized":"2024-04-15T15:15:15.865159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311080+0000","last_change":"2024-04-15T15:15:14.323489+0000","last_active":"2024-04-15T15:15:15.311080+0000","last_peered":"2024-04-15T15:15:15.311080+0000","last_clean":"2024-04-15T15:15:15.311080+0000","last_became_active":"2024-04-15T15:15:14.322956+0000","last_became_peered":"2024-04-15T15:15:14.322956+0000","last_unstale":"2024-04-15T15:15:15.311080+0000","last_undegraded":"2024-04-15T15:15:15.311080+0000","last_fullsized":"2024-04-15T15:15:15.311080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308115+0000","last_change":"2024-04-15T15:15:10.295446+0000","last_active":"2024-04-15T15:15:15.308115+0000","last_peered":"2024-04-15T15:15:15.308115+0000","last_clean":"2024-04-15T15:15:15.308115+0000","last_became_active":"2024-04-15T15:15:10.295165+0000","last_became_peered":"2024-04-15T15:15:10.295165+0000","last_unstale":"2024-04-15T15:15:15.308115+0000","last_undegraded":"2024-04-15T15:15:15.308115+0000","last_fullsized":"2024-04-15T15:15:15.308115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308300+0000","last_change":"2024-04-15T15:15:09.289277+0000","last_active":"2024-04-15T15:15:15.308300+0000","last_peered":"2024-04-15T15:15:15.308300+0000","last_clean":"2024-04-15T15:15:15.308300+0000","last_became_active":"2024-04-15T15:15:09.289026+0000","last_became_peered":"2024-04-15T15:15:09.289026+0000","last_unstale":"2024-04-15T15:15:15.308300+0000","last_undegraded":"2024-04-15T15:15:15.308300+0000","last_fullsized":"2024-04-15T15:15:15.308300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310503+0000","last_change":"2024-04-15T15:15:09.285504+0000","last_active":"2024-04-15T15:15:15.310503+0000","last_peered":"2024-04-15T15:15:15.310503+0000","last_clean":"2024-04-15T15:15:15.310503+0000","last_became_active":"2024-04-15T15:15:09.285219+0000","last_became_peered":"2024-04-15T15:15:09.285219+0000","last_unstale":"2024-04-15T15:15:15.310503+0000","last_undegraded":"2024-04-15T15:15:15.310503+0000","last_fullsized":"2024-04-15T15:15:15.310503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865651+0000","last_change":"2024-04-15T15:15:10.297720+0000","last_active":"2024-04-15T15:15:15.865651+0000","last_peered":"2024-04-15T15:15:15.865651+0000","last_clean":"2024-04-15T15:15:15.865651+0000","last_became_active":"2024-04-15T15:15:10.297469+0000","last_became_peered":"2024-04-15T15:15:10.297469+0000","last_unstale":"2024-04-15T15:15:15.865651+0000","last_undegraded":"2024-04-15T15:15:15.865651+0000","last_fullsized":"2024-04-15T15:15:15.865651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864649+0000","last_change":"2024-04-15T15:15:14.327342+0000","last_active":"2024-04-15T15:15:15.864649+0000","last_peered":"2024-04-15T15:15:15.864649+0000","last_clean":"2024-04-15T15:15:15.864649+0000","last_became_active":"2024-04-15T15:15:14.326929+0000","last_became_peered":"2024-04-15T15:15:14.326929+0000","last_unstale":"2024-04-15T15:15:15.864649+0000","last_undegraded":"2024-04-15T15:15:15.864649+0000","last_fullsized":"2024-04-15T15:15:15.864649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:19:56.492223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308298+0000","last_change":"2024-04-15T15:15:09.288401+0000","last_active":"2024-04-15T15:15:15.308298+0000","last_peered":"2024-04-15T15:15:15.308298+0000","last_clean":"2024-04-15T15:15:15.308298+0000","last_became_active":"2024-04-15T15:15:09.288124+0000","last_became_peered":"2024-04-15T15:15:09.288124+0000","last_unstale":"2024-04-15T15:15:15.308298+0000","last_undegraded":"2024-04-15T15:15:15.308298+0000","last_fullsized":"2024-04-15T15:15:15.308298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309493+0000","last_change":"2024-04-15T15:15:10.293787+0000","last_active":"2024-04-15T15:15:15.309493+0000","last_peered":"2024-04-15T15:15:15.309493+0000","last_clean":"2024-04-15T15:15:15.309493+0000","last_became_active":"2024-04-15T15:15:10.293409+0000","last_became_peered":"2024-04-15T15:15:10.293409+0000","last_unstale":"2024-04-15T15:15:15.309493+0000","last_undegraded":"2024-04-15T15:15:15.309493+0000","last_fullsized":"2024-04-15T15:15:15.309493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:30.764771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308821+0000","last_change":"2024-04-15T15:15:14.323853+0000","last_active":"2024-04-15T15:15:15.308821+0000","last_peered":"2024-04-15T15:15:15.308821+0000","last_clean":"2024-04-15T15:15:15.308821+0000","last_became_active":"2024-04-15T15:15:14.323547+0000","last_became_peered":"2024-04-15T15:15:14.323547+0000","last_unstale":"2024-04-15T15:15:15.308821+0000","last_undegraded":"2024-04-15T15:15:15.308821+0000","last_fullsized":"2024-04-15T15:15:15.308821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:48.237751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308815+0000","last_change":"2024-04-15T15:15:10.290641+0000","last_active":"2024-04-15T15:15:15.308815+0000","last_peered":"2024-04-15T15:15:15.308815+0000","last_clean":"2024-04-15T15:15:15.308815+0000","last_became_active":"2024-04-15T15:15:10.290342+0000","last_became_peered":"2024-04-15T15:15:10.290342+0000","last_unstale":"2024-04-15T15:15:15.308815+0000","last_undegraded":"2024-04-15T15:15:15.308815+0000","last_fullsized":"2024-04-15T15:15:15.308815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864807+0000","last_change":"2024-04-15T15:15:10.302121+0000","last_active":"2024-04-15T15:15:15.864807+0000","last_peered":"2024-04-15T15:15:15.864807+0000","last_clean":"2024-04-15T15:15:15.864807+0000","last_became_active":"2024-04-15T15:15:10.301860+0000","last_became_peered":"2024-04-15T15:15:10.301860+0000","last_unstale":"2024-04-15T15:15:15.864807+0000","last_undegraded":"2024-04-15T15:15:15.864807+0000","last_fullsized":"2024-04-15T15:15:15.864807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865207+0000","last_change":"2024-04-15T15:15:10.298052+0000","last_active":"2024-04-15T15:15:15.865207+0000","last_peered":"2024-04-15T15:15:15.865207+0000","last_clean":"2024-04-15T15:15:15.865207+0000","last_became_active":"2024-04-15T15:15:10.296656+0000","last_became_peered":"2024-04-15T15:15:10.296656+0000","last_unstale":"2024-04-15T15:15:15.865207+0000","last_undegraded":"2024-04-15T15:15:15.865207+0000","last_fullsized":"2024-04-15T15:15:15.865207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865604+0000","last_change":"2024-04-15T15:15:10.296027+0000","last_active":"2024-04-15T15:15:15.865604+0000","last_peered":"2024-04-15T15:15:15.865604+0000","last_clean":"2024-04-15T15:15:15.865604+0000","last_became_active":"2024-04-15T15:15:10.295741+0000","last_became_peered":"2024-04-15T15:15:10.295741+0000","last_unstale":"2024-04-15T15:15:15.865604+0000","last_undegraded":"2024-04-15T15:15:15.865604+0000","last_fullsized":"2024-04-15T15:15:15.865604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308554+0000","last_change":"2024-04-15T15:15:10.287343+0000","last_active":"2024-04-15T15:15:15.308554+0000","last_peered":"2024-04-15T15:15:15.308554+0000","last_clean":"2024-04-15T15:15:15.308554+0000","last_became_active":"2024-04-15T15:15:10.287082+0000","last_became_peered":"2024-04-15T15:15:10.287082+0000","last_unstale":"2024-04-15T15:15:15.308554+0000","last_undegraded":"2024-04-15T15:15:15.308554+0000","last_fullsized":"2024-04-15T15:15:15.308554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393974+0000","last_change":"2024-04-15T15:15:10.289020+0000","last_active":"2024-04-15T15:15:15.393974+0000","last_peered":"2024-04-15T15:15:15.393974+0000","last_clean":"2024-04-15T15:15:15.393974+0000","last_became_active":"2024-04-15T15:15:10.288777+0000","last_became_peered":"2024-04-15T15:15:10.288777+0000","last_unstale":"2024-04-15T15:15:15.393974+0000","last_undegraded":"2024-04-15T15:15:15.393974+0000","last_fullsized":"2024-04-15T15:15:15.393974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310906+0000","last_change":"2024-04-15T15:15:10.290594+0000","last_active":"2024-04-15T15:15:15.310906+0000","last_peered":"2024-04-15T15:15:15.310906+0000","last_clean":"2024-04-15T15:15:15.310906+0000","last_became_active":"2024-04-15T15:15:10.290345+0000","last_became_peered":"2024-04-15T15:15:10.290345+0000","last_unstale":"2024-04-15T15:15:15.310906+0000","last_undegraded":"2024-04-15T15:15:15.310906+0000","last_fullsized":"2024-04-15T15:15:15.310906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308747+0000","last_change":"2024-04-15T15:15:10.297500+0000","last_active":"2024-04-15T15:15:15.308747+0000","last_peered":"2024-04-15T15:15:15.308747+0000","last_clean":"2024-04-15T15:15:15.308747+0000","last_became_active":"2024-04-15T15:15:10.296730+0000","last_became_peered":"2024-04-15T15:15:10.296730+0000","last_unstale":"2024-04-15T15:15:15.308747+0000","last_undegraded":"2024-04-15T15:15:15.308747+0000","last_fullsized":"2024-04-15T15:15:15.308747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865052+0000","last_change":"2024-04-15T15:15:10.296225+0000","last_active":"2024-04-15T15:15:15.865052+0000","last_peered":"2024-04-15T15:15:15.865052+0000","last_clean":"2024-04-15T15:15:15.865052+0000","last_became_active":"2024-04-15T15:15:10.295930+0000","last_became_peered":"2024-04-15T15:15:10.295930+0000","last_unstale":"2024-04-15T15:15:15.865052+0000","last_undegraded":"2024-04-15T15:15:15.865052+0000","last_fullsized":"2024-04-15T15:15:15.865052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308196+0000","last_change":"2024-04-15T15:15:10.289427+0000","last_active":"2024-04-15T15:15:15.308196+0000","last_peered":"2024-04-15T15:15:15.308196+0000","last_clean":"2024-04-15T15:15:15.308196+0000","last_became_active":"2024-04-15T15:15:10.289170+0000","last_became_peered":"2024-04-15T15:15:10.289170+0000","last_unstale":"2024-04-15T15:15:15.308196+0000","last_undegraded":"2024-04-15T15:15:15.308196+0000","last_fullsized":"2024-04-15T15:15:15.308196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865103+0000","last_change":"2024-04-15T15:15:10.304028+0000","last_active":"2024-04-15T15:15:15.865103+0000","last_peered":"2024-04-15T15:15:15.865103+0000","last_clean":"2024-04-15T15:15:15.865103+0000","last_became_active":"2024-04-15T15:15:10.303771+0000","last_became_peered":"2024-04-15T15:15:10.303771+0000","last_unstale":"2024-04-15T15:15:15.865103+0000","last_undegraded":"2024-04-15T15:15:15.865103+0000","last_fullsized":"2024-04-15T15:15:15.865103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865533+0000","last_change":"2024-04-15T15:15:10.293639+0000","last_active":"2024-04-15T15:15:15.865533+0000","last_peered":"2024-04-15T15:15:15.865533+0000","last_clean":"2024-04-15T15:15:15.865533+0000","last_became_active":"2024-04-15T15:15:10.293220+0000","last_became_peered":"2024-04-15T15:15:10.293220+0000","last_unstale":"2024-04-15T15:15:15.865533+0000","last_undegraded":"2024-04-15T15:15:15.865533+0000","last_fullsized":"2024-04-15T15:15:15.865533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310554+0000","last_change":"2024-04-15T15:15:10.289725+0000","last_active":"2024-04-15T15:15:15.310554+0000","last_peered":"2024-04-15T15:15:15.310554+0000","last_clean":"2024-04-15T15:15:15.310554+0000","last_became_active":"2024-04-15T15:15:10.289378+0000","last_became_peered":"2024-04-15T15:15:10.289378+0000","last_unstale":"2024-04-15T15:15:15.310554+0000","last_undegraded":"2024-04-15T15:15:15.310554+0000","last_fullsized":"2024-04-15T15:15:15.310554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866173+0000","last_change":"2024-04-15T15:15:10.295231+0000","last_active":"2024-04-15T15:15:15.866173+0000","last_peered":"2024-04-15T15:15:15.866173+0000","last_clean":"2024-04-15T15:15:15.866173+0000","last_became_active":"2024-04-15T15:15:10.294974+0000","last_became_peered":"2024-04-15T15:15:10.294974+0000","last_unstale":"2024-04-15T15:15:15.866173+0000","last_undegraded":"2024-04-15T15:15:15.866173+0000","last_fullsized":"2024-04-15T15:15:15.866173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864735+0000","last_change":"2024-04-15T15:15:10.300143+0000","last_active":"2024-04-15T15:15:15.864735+0000","last_peered":"2024-04-15T15:15:15.864735+0000","last_clean":"2024-04-15T15:15:15.864735+0000","last_became_active":"2024-04-15T15:15:10.299889+0000","last_became_peered":"2024-04-15T15:15:10.299889+0000","last_unstale":"2024-04-15T15:15:15.864735+0000","last_undegraded":"2024-04-15T15:15:15.864735+0000","last_fullsized":"2024-04-15T15:15:15.864735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:46.735175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865763+0000","last_change":"2024-04-15T15:15:10.295205+0000","last_active":"2024-04-15T15:15:15.865763+0000","last_peered":"2024-04-15T15:15:15.865763+0000","last_clean":"2024-04-15T15:15:15.865763+0000","last_became_active":"2024-04-15T15:15:10.294556+0000","last_became_peered":"2024-04-15T15:15:10.294556+0000","last_unstale":"2024-04-15T15:15:15.865763+0000","last_undegraded":"2024-04-15T15:15:15.865763+0000","last_fullsized":"2024-04-15T15:15:15.865763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864921+0000","last_change":"2024-04-15T15:15:10.297025+0000","last_active":"2024-04-15T15:15:15.864921+0000","last_peered":"2024-04-15T15:15:15.864921+0000","last_clean":"2024-04-15T15:15:15.864921+0000","last_became_active":"2024-04-15T15:15:10.296519+0000","last_became_peered":"2024-04-15T15:15:10.296519+0000","last_unstale":"2024-04-15T15:15:15.864921+0000","last_undegraded":"2024-04-15T15:15:15.864921+0000","last_fullsized":"2024-04-15T15:15:15.864921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393530+0000","last_change":"2024-04-15T15:15:10.288655+0000","last_active":"2024-04-15T15:15:15.393530+0000","last_peered":"2024-04-15T15:15:15.393530+0000","last_clean":"2024-04-15T15:15:15.393530+0000","last_became_active":"2024-04-15T15:15:10.286513+0000","last_became_peered":"2024-04-15T15:15:10.286513+0000","last_unstale":"2024-04-15T15:15:15.393530+0000","last_undegraded":"2024-04-15T15:15:15.393530+0000","last_fullsized":"2024-04-15T15:15:15.393530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865152+0000","last_change":"2024-04-15T15:15:10.299829+0000","last_active":"2024-04-15T15:15:15.865152+0000","last_peered":"2024-04-15T15:15:15.865152+0000","last_clean":"2024-04-15T15:15:15.865152+0000","last_became_active":"2024-04-15T15:15:10.299281+0000","last_became_peered":"2024-04-15T15:15:10.299281+0000","last_unstale":"2024-04-15T15:15:15.865152+0000","last_undegraded":"2024-04-15T15:15:15.865152+0000","last_fullsized":"2024-04-15T15:15:15.865152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:26:12.807802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308094+0000","last_change":"2024-04-15T15:15:10.301135+0000","last_active":"2024-04-15T15:15:15.308094+0000","last_peered":"2024-04-15T15:15:15.308094+0000","last_clean":"2024-04-15T15:15:15.308094+0000","last_became_active":"2024-04-15T15:15:10.300869+0000","last_became_peered":"2024-04-15T15:15:10.300869+0000","last_unstale":"2024-04-15T15:15:15.308094+0000","last_undegraded":"2024-04-15T15:15:15.308094+0000","last_fullsized":"2024-04-15T15:15:15.308094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309542+0000","last_change":"2024-04-15T15:15:10.285169+0000","last_active":"2024-04-15T15:15:15.309542+0000","last_peered":"2024-04-15T15:15:15.309542+0000","last_clean":"2024-04-15T15:15:15.309542+0000","last_became_active":"2024-04-15T15:15:10.284906+0000","last_became_peered":"2024-04-15T15:15:10.284906+0000","last_unstale":"2024-04-15T15:15:15.309542+0000","last_undegraded":"2024-04-15T15:15:15.309542+0000","last_fullsized":"2024-04-15T15:15:15.309542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:18.325913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310364+0000","last_change":"2024-04-15T15:15:10.300827+0000","last_active":"2024-04-15T15:15:15.310364+0000","last_peered":"2024-04-15T15:15:15.310364+0000","last_clean":"2024-04-15T15:15:15.310364+0000","last_became_active":"2024-04-15T15:15:10.300530+0000","last_became_peered":"2024-04-15T15:15:10.300530+0000","last_unstale":"2024-04-15T15:15:15.310364+0000","last_undegraded":"2024-04-15T15:15:15.310364+0000","last_fullsized":"2024-04-15T15:15:15.310364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310951+0000","last_change":"2024-04-15T15:15:10.297517+0000","last_active":"2024-04-15T15:15:15.310951+0000","last_peered":"2024-04-15T15:15:15.310951+0000","last_clean":"2024-04-15T15:15:15.310951+0000","last_became_active":"2024-04-15T15:15:10.297224+0000","last_became_peered":"2024-04-15T15:15:10.297224+0000","last_unstale":"2024-04-15T15:15:15.310951+0000","last_undegraded":"2024-04-15T15:15:15.310951+0000","last_fullsized":"2024-04-15T15:15:15.310951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866349+0000","last_change":"2024-04-15T15:15:10.305356+0000","last_active":"2024-04-15T15:15:15.866349+0000","last_peered":"2024-04-15T15:15:15.866349+0000","last_clean":"2024-04-15T15:15:15.866349+0000","last_became_active":"2024-04-15T15:15:10.305035+0000","last_became_peered":"2024-04-15T15:15:10.305035+0000","last_unstale":"2024-04-15T15:15:15.866349+0000","last_undegraded":"2024-04-15T15:15:15.866349+0000","last_fullsized":"2024-04-15T15:15:15.866349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:00:23.860901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308001+0000","last_change":"2024-04-15T15:15:10.299528+0000","last_active":"2024-04-15T15:15:15.308001+0000","last_peered":"2024-04-15T15:15:15.308001+0000","last_clean":"2024-04-15T15:15:15.308001+0000","last_became_active":"2024-04-15T15:15:10.299272+0000","last_became_peered":"2024-04-15T15:15:10.299272+0000","last_unstale":"2024-04-15T15:15:15.308001+0000","last_undegraded":"2024-04-15T15:15:15.308001+0000","last_fullsized":"2024-04-15T15:15:15.308001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865187+0000","last_change":"2024-04-15T15:15:10.295288+0000","last_active":"2024-04-15T15:15:15.865187+0000","last_peered":"2024-04-15T15:15:15.865187+0000","last_clean":"2024-04-15T15:15:15.865187+0000","last_became_active":"2024-04-15T15:15:10.295000+0000","last_became_peered":"2024-04-15T15:15:10.295000+0000","last_unstale":"2024-04-15T15:15:15.865187+0000","last_undegraded":"2024-04-15T15:15:15.865187+0000","last_fullsized":"2024-04-15T15:15:15.865187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864972+0000","last_change":"2024-04-15T15:15:10.298860+0000","last_active":"2024-04-15T15:15:15.864972+0000","last_peered":"2024-04-15T15:15:15.864972+0000","last_clean":"2024-04-15T15:15:15.864972+0000","last_became_active":"2024-04-15T15:15:10.296935+0000","last_became_peered":"2024-04-15T15:15:10.296935+0000","last_unstale":"2024-04-15T15:15:15.864972+0000","last_undegraded":"2024-04-15T15:15:15.864972+0000","last_fullsized":"2024-04-15T15:15:15.864972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865016+0000","last_change":"2024-04-15T15:15:10.296047+0000","last_active":"2024-04-15T15:15:15.865016+0000","last_peered":"2024-04-15T15:15:15.865016+0000","last_clean":"2024-04-15T15:15:15.865016+0000","last_became_active":"2024-04-15T15:15:10.295590+0000","last_became_peered":"2024-04-15T15:15:10.295590+0000","last_unstale":"2024-04-15T15:15:15.865016+0000","last_undegraded":"2024-04-15T15:15:15.865016+0000","last_fullsized":"2024-04-15T15:15:15.865016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865557+0000","last_change":"2024-04-15T15:15:10.298993+0000","last_active":"2024-04-15T15:15:15.865557+0000","last_peered":"2024-04-15T15:15:15.865557+0000","last_clean":"2024-04-15T15:15:15.865557+0000","last_became_active":"2024-04-15T15:15:10.298652+0000","last_became_peered":"2024-04-15T15:15:10.298652+0000","last_unstale":"2024-04-15T15:15:15.865557+0000","last_undegraded":"2024-04-15T15:15:15.865557+0000","last_fullsized":"2024-04-15T15:15:15.865557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865100+0000","last_change":"2024-04-15T15:15:10.291981+0000","last_active":"2024-04-15T15:15:15.865100+0000","last_peered":"2024-04-15T15:15:15.865100+0000","last_clean":"2024-04-15T15:15:15.865100+0000","last_became_active":"2024-04-15T15:15:10.291495+0000","last_became_peered":"2024-04-15T15:15:10.291495+0000","last_unstale":"2024-04-15T15:15:15.865100+0000","last_undegraded":"2024-04-15T15:15:15.865100+0000","last_fullsized":"2024-04-15T15:15:15.865100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394232+0000","last_change":"2024-04-15T15:15:10.288087+0000","last_active":"2024-04-15T15:15:15.394232+0000","last_peered":"2024-04-15T15:15:15.394232+0000","last_clean":"2024-04-15T15:15:15.394232+0000","last_became_active":"2024-04-15T15:15:10.287803+0000","last_became_peered":"2024-04-15T15:15:10.287803+0000","last_unstale":"2024-04-15T15:15:15.394232+0000","last_undegraded":"2024-04-15T15:15:15.394232+0000","last_fullsized":"2024-04-15T15:15:15.394232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866222+0000","last_change":"2024-04-15T15:15:10.302148+0000","last_active":"2024-04-15T15:15:15.866222+0000","last_peered":"2024-04-15T15:15:15.866222+0000","last_clean":"2024-04-15T15:15:15.866222+0000","last_became_active":"2024-04-15T15:15:10.301882+0000","last_became_peered":"2024-04-15T15:15:10.301882+0000","last_unstale":"2024-04-15T15:15:15.866222+0000","last_undegraded":"2024-04-15T15:15:15.866222+0000","last_fullsized":"2024-04-15T15:15:15.866222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":104773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":104807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":695653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:31.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:15:31.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.908+0000 7f7178cba640 1 Processor -- start 2024-04-15T15:15:31.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.908+0000 7f7178cba640 1 -- start start 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7178cba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f717415c660 0x7f717406e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7178cba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7178cba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f717406eaf0 con 0x7f717415cf80 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7178cba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f717406edd0 con 0x7f717415c660 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7178cba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f717406f0b0 con 0x7f71740ccb20 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f717259c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f717415c660 0x7f717406e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:43162/0 (socket says 172.21.15.70:43162) 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 -- 172.21.15.70:0/3197585894 learned_addr learned my addr 172.21.15.70:0/3197585894 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 -- 172.21.15.70:0/3197585894 >> v2:172.21.15.96:3300/0 conn(0x7f717415c660 msgr2=0x7f717406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 --2- 172.21.15.70:0/3197585894 >> v2:172.21.15.96:3300/0 conn(0x7f717415c660 0x7f717406e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 -- 172.21.15.70:0/3197585894 >> v1:172.21.15.96:6789/0 conn(0x7f717415cf80 legacy=0x7f71740cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.909+0000 7f7169d9b640 1 -- 172.21.15.70:0/3197585894 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717406f390 con 0x7f71740ccb20 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.910+0000 7f7169d9b640 1 --2- 172.21.15.70:0/3197585894 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f715c009870 tx=0x7f715c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=765f6abe0b35d696 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.910+0000 7f7171d9b640 1 -- 172.21.15.70:0/3197585894 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c035020 con 0x7f71740ccb20 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.910+0000 7f7171d9b640 1 -- 172.21.15.70:0/3197585894 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f715c031640 con 0x7f71740ccb20 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.910+0000 7f7171d9b640 1 -- 172.21.15.70:0/3197585894 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f715c02d040 con 0x7f71740ccb20 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 -- 172.21.15.70:0/3197585894 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 msgr2=0x7f717415ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 --2- 172.21.15.70:0/3197585894 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f715c009870 tx=0x7f715c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 -- 172.21.15.70:0/3197585894 shutdown_connections 2024-04-15T15:15:31.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 --2- 172.21.15.70:0/3197585894 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f71740ccb20 0x7f717415ca40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 --2- 172.21.15.70:0/3197585894 >> v2:172.21.15.96:3300/0 conn(0x7f717415c660 0x7f717406e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 -- 172.21.15.70:0/3197585894 >> 172.21.15.70:0/3197585894 conn(0x7f717406b9e0 msgr2=0x7f717406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:31.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 -- 172.21.15.70:0/3197585894 shutdown_connections 2024-04-15T15:15:31.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.911+0000 7f7178cba640 1 -- 172.21.15.70:0/3197585894 wait complete. 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.912+0000 7f7178cba640 1 Processor -- start 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.912+0000 7f7178cba640 1 -- start start 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f717415c660 0x7f717414fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f717415cf80 0x7f71741500f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f71740d0180 con 0x7f71740ccb20 2024-04-15T15:15:31.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f71740d1110 con 0x7f717415c660 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7178cba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f71740d0300 con 0x7f717415cf80 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36894/0 (socket says 172.21.15.70:36894) 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 -- 172.21.15.70:0/2418303126 learned_addr learned my addr 172.21.15.70:0/2418303126 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7169d9b640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f717415c660 0x7f717414fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f7172d9d640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f717415cf80 0x7f71741500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 -- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f717415cf80 msgr2=0x7f71741500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f717415cf80 0x7f71741500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 -- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f717415c660 msgr2=0x7f717414fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.913+0000 7f717259c640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f717415c660 0x7f717414fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f717259c640 1 -- 172.21.15.70:0/2418303126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f717414e810 con 0x7f71740ccb20 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f717259c640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f716400e8a0 tx=0x7f716400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164017070 con 0x7f71740ccb20 2024-04-15T15:15:31.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716400c940 con 0x7f71740ccb20 2024-04-15T15:15:31.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716400cd30 con 0x7f71740ccb20 2024-04-15T15:15:31.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7174150720 con 0x7f71740ccb20 2024-04-15T15:15:31.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.914+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7174156350 con 0x7f71740ccb20 2024-04-15T15:15:31.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.915+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71740689e0 con 0x7f71740ccb20 2024-04-15T15:15:31.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.915+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f716401a070 con 0x7f71740ccb20 2024-04-15T15:15:31.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.916+0000 7f716b7fe640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 0x7f715406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:31.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.916+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f716408c020 con 0x7f71740ccb20 2024-04-15T15:15:31.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.916+0000 7f7169d9b640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 0x7f715406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:31.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.916+0000 7f7169d9b640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 0x7f715406edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f715c036000 tx=0x7f715c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:31.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:31.920+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f717414e810 con 0x7f71740ccb20 2024-04-15T15:15:32.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.093+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f717415a810 con 0x7f715406c8f0 2024-04-15T15:15:32.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.100+0000 7f716b7fe640 1 -- 172.21.15.70:0/2418303126 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f715c099310 con 0x7f715406c8f0 2024-04-15T15:15:32.105 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:32.107 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.103+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 msgr2=0x7f715406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.103+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 0x7f715406edb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f715c036000 tx=0x7f715c03c040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 msgr2=0x7f717414e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f716400e8a0 tx=0x7f716400ed60 comp rx=0 tx=0).stop 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 shutdown_connections 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f717415cf80 0x7f71741500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f717415c660 0x7f717414fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f715406c8f0 0x7f715406edb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 --2- 172.21.15.70:0/2418303126 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f71740ccb20 0x7f717414e2d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 >> 172.21.15.70:0/2418303126 conn(0x7f717406b9e0 msgr2=0x7f717415af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 shutdown_connections 2024-04-15T15:15:32.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.104+0000 7f7178cba640 1 -- 172.21.15.70:0/2418303126 wait complete. 2024-04-15T15:15:32.117 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-04-15T15:15:32.074623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236128,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738592,"statfs":{"total":773094113280,"available":772852318208,"internally_reserved":0,"allocated":3383296,"data_stored":2027192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002619"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":26,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865551+0000","last_change":"2024-04-15T15:15:09.296489+0000","last_active":"2024-04-15T15:15:15.865551+0000","last_peered":"2024-04-15T15:15:15.865551+0000","last_clean":"2024-04-15T15:15:15.865551+0000","last_became_active":"2024-04-15T15:15:09.296225+0000","last_became_peered":"2024-04-15T15:15:09.296225+0000","last_unstale":"2024-04-15T15:15:15.865551+0000","last_undegraded":"2024-04-15T15:15:15.865551+0000","last_fullsized":"2024-04-15T15:15:15.865551+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308730+0000","last_change":"2024-04-15T15:15:10.297175+0000","last_active":"2024-04-15T15:15:15.308730+0000","last_peered":"2024-04-15T15:15:15.308730+0000","last_clean":"2024-04-15T15:15:15.308730+0000","last_became_active":"2024-04-15T15:15:10.296707+0000","last_became_peered":"2024-04-15T15:15:10.296707+0000","last_unstale":"2024-04-15T15:15:15.308730+0000","last_undegraded":"2024-04-15T15:15:15.308730+0000","last_fullsized":"2024-04-15T15:15:15.308730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308030+0000","last_change":"2024-04-15T15:15:14.490094+0000","last_active":"2024-04-15T15:15:15.308030+0000","last_peered":"2024-04-15T15:15:15.308030+0000","last_clean":"2024-04-15T15:15:15.308030+0000","last_became_active":"2024-04-15T15:15:14.489727+0000","last_became_peered":"2024-04-15T15:15:14.489727+0000","last_unstale":"2024-04-15T15:15:15.308030+0000","last_undegraded":"2024-04-15T15:15:15.308030+0000","last_fullsized":"2024-04-15T15:15:15.308030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308822+0000","last_change":"2024-04-15T15:15:09.279298+0000","last_active":"2024-04-15T15:15:15.308822+0000","last_peered":"2024-04-15T15:15:15.308822+0000","last_clean":"2024-04-15T15:15:15.308822+0000","last_became_active":"2024-04-15T15:15:09.279018+0000","last_became_peered":"2024-04-15T15:15:09.279018+0000","last_unstale":"2024-04-15T15:15:15.308822+0000","last_undegraded":"2024-04-15T15:15:15.308822+0000","last_fullsized":"2024-04-15T15:15:15.308822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865283+0000","last_change":"2024-04-15T15:15:10.300836+0000","last_active":"2024-04-15T15:15:15.865283+0000","last_peered":"2024-04-15T15:15:15.865283+0000","last_clean":"2024-04-15T15:15:15.865283+0000","last_became_active":"2024-04-15T15:15:10.300574+0000","last_became_peered":"2024-04-15T15:15:10.300574+0000","last_unstale":"2024-04-15T15:15:15.865283+0000","last_undegraded":"2024-04-15T15:15:15.865283+0000","last_fullsized":"2024-04-15T15:15:15.865283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310208+0000","last_change":"2024-04-15T15:15:14.327665+0000","last_active":"2024-04-15T15:15:15.310208+0000","last_peered":"2024-04-15T15:15:15.310208+0000","last_clean":"2024-04-15T15:15:15.310208+0000","last_became_active":"2024-04-15T15:15:14.327161+0000","last_became_peered":"2024-04-15T15:15:14.327161+0000","last_unstale":"2024-04-15T15:15:15.310208+0000","last_undegraded":"2024-04-15T15:15:15.310208+0000","last_fullsized":"2024-04-15T15:15:15.310208+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866087+0000","last_change":"2024-04-15T15:15:09.297431+0000","last_active":"2024-04-15T15:15:15.866087+0000","last_peered":"2024-04-15T15:15:15.866087+0000","last_clean":"2024-04-15T15:15:15.866087+0000","last_became_active":"2024-04-15T15:15:09.297182+0000","last_became_peered":"2024-04-15T15:15:09.297182+0000","last_unstale":"2024-04-15T15:15:15.866087+0000","last_undegraded":"2024-04-15T15:15:15.866087+0000","last_fullsized":"2024-04-15T15:15:15.866087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:45.731243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865971+0000","last_change":"2024-04-15T15:15:10.303063+0000","last_active":"2024-04-15T15:15:15.865971+0000","last_peered":"2024-04-15T15:15:15.865971+0000","last_clean":"2024-04-15T15:15:15.865971+0000","last_became_active":"2024-04-15T15:15:10.302785+0000","last_became_peered":"2024-04-15T15:15:10.302785+0000","last_unstale":"2024-04-15T15:15:15.865971+0000","last_undegraded":"2024-04-15T15:15:15.865971+0000","last_fullsized":"2024-04-15T15:15:15.865971+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309364+0000","last_change":"2024-04-15T15:15:14.328804+0000","last_active":"2024-04-15T15:15:15.309364+0000","last_peered":"2024-04-15T15:15:15.309364+0000","last_clean":"2024-04-15T15:15:15.309364+0000","last_became_active":"2024-04-15T15:15:14.327886+0000","last_became_peered":"2024-04-15T15:15:14.327886+0000","last_unstale":"2024-04-15T15:15:15.309364+0000","last_undegraded":"2024-04-15T15:15:15.309364+0000","last_fullsized":"2024-04-15T15:15:15.309364+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:25.064499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865605+0000","last_change":"2024-04-15T15:15:09.300461+0000","last_active":"2024-04-15T15:15:15.865605+0000","last_peered":"2024-04-15T15:15:15.865605+0000","last_clean":"2024-04-15T15:15:15.865605+0000","last_became_active":"2024-04-15T15:15:09.300159+0000","last_became_peered":"2024-04-15T15:15:09.300159+0000","last_unstale":"2024-04-15T15:15:15.865605+0000","last_undegraded":"2024-04-15T15:15:15.865605+0000","last_fullsized":"2024-04-15T15:15:15.865605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865784+0000","last_change":"2024-04-15T15:15:10.298282+0000","last_active":"2024-04-15T15:15:15.865784+0000","last_peered":"2024-04-15T15:15:15.865784+0000","last_clean":"2024-04-15T15:15:15.865784+0000","last_became_active":"2024-04-15T15:15:10.297734+0000","last_became_peered":"2024-04-15T15:15:10.297734+0000","last_unstale":"2024-04-15T15:15:15.865784+0000","last_undegraded":"2024-04-15T15:15:15.865784+0000","last_fullsized":"2024-04-15T15:15:15.865784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309722+0000","last_change":"2024-04-15T15:15:14.321155+0000","last_active":"2024-04-15T15:15:15.309722+0000","last_peered":"2024-04-15T15:15:15.309722+0000","last_clean":"2024-04-15T15:15:15.309722+0000","last_became_active":"2024-04-15T15:15:14.320777+0000","last_became_peered":"2024-04-15T15:15:14.320777+0000","last_unstale":"2024-04-15T15:15:15.309722+0000","last_undegraded":"2024-04-15T15:15:15.309722+0000","last_fullsized":"2024-04-15T15:15:15.309722+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:36.915564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310304+0000","last_change":"2024-04-15T15:15:10.288957+0000","last_active":"2024-04-15T15:15:15.310304+0000","last_peered":"2024-04-15T15:15:15.310304+0000","last_clean":"2024-04-15T15:15:15.310304+0000","last_became_active":"2024-04-15T15:15:10.288692+0000","last_became_peered":"2024-04-15T15:15:10.288692+0000","last_unstale":"2024-04-15T15:15:15.310304+0000","last_undegraded":"2024-04-15T15:15:15.310304+0000","last_fullsized":"2024-04-15T15:15:15.310304+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865797+0000","last_change":"2024-04-15T15:15:09.295590+0000","last_active":"2024-04-15T15:15:15.865797+0000","last_peered":"2024-04-15T15:15:15.865797+0000","last_clean":"2024-04-15T15:15:15.865797+0000","last_became_active":"2024-04-15T15:15:09.295295+0000","last_became_peered":"2024-04-15T15:15:09.295295+0000","last_unstale":"2024-04-15T15:15:15.865797+0000","last_undegraded":"2024-04-15T15:15:15.865797+0000","last_fullsized":"2024-04-15T15:15:15.865797+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309855+0000","last_change":"2024-04-15T15:15:14.327835+0000","last_active":"2024-04-15T15:15:15.309855+0000","last_peered":"2024-04-15T15:15:15.309855+0000","last_clean":"2024-04-15T15:15:15.309855+0000","last_became_active":"2024-04-15T15:15:14.327245+0000","last_became_peered":"2024-04-15T15:15:14.327245+0000","last_unstale":"2024-04-15T15:15:15.309855+0000","last_undegraded":"2024-04-15T15:15:15.309855+0000","last_fullsized":"2024-04-15T15:15:15.309855+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308188+0000","last_change":"2024-04-15T15:15:10.295527+0000","last_active":"2024-04-15T15:15:15.308188+0000","last_peered":"2024-04-15T15:15:15.308188+0000","last_clean":"2024-04-15T15:15:15.308188+0000","last_became_active":"2024-04-15T15:15:10.295265+0000","last_became_peered":"2024-04-15T15:15:10.295265+0000","last_unstale":"2024-04-15T15:15:15.308188+0000","last_undegraded":"2024-04-15T15:15:15.308188+0000","last_fullsized":"2024-04-15T15:15:15.308188+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393749+0000","last_change":"2024-04-15T15:15:09.291151+0000","last_active":"2024-04-15T15:15:15.393749+0000","last_peered":"2024-04-15T15:15:15.393749+0000","last_clean":"2024-04-15T15:15:15.393749+0000","last_became_active":"2024-04-15T15:15:09.290851+0000","last_became_peered":"2024-04-15T15:15:09.290851+0000","last_unstale":"2024-04-15T15:15:15.393749+0000","last_undegraded":"2024-04-15T15:15:15.393749+0000","last_fullsized":"2024-04-15T15:15:15.393749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309375+0000","last_change":"2024-04-15T15:15:14.327866+0000","last_active":"2024-04-15T15:15:15.309375+0000","last_peered":"2024-04-15T15:15:15.309375+0000","last_clean":"2024-04-15T15:15:15.309375+0000","last_became_active":"2024-04-15T15:15:14.327398+0000","last_became_peered":"2024-04-15T15:15:14.327398+0000","last_unstale":"2024-04-15T15:15:15.309375+0000","last_undegraded":"2024-04-15T15:15:15.309375+0000","last_fullsized":"2024-04-15T15:15:15.309375+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:45.919286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308185+0000","last_change":"2024-04-15T15:15:10.294406+0000","last_active":"2024-04-15T15:15:15.308185+0000","last_peered":"2024-04-15T15:15:15.308185+0000","last_clean":"2024-04-15T15:15:15.308185+0000","last_became_active":"2024-04-15T15:15:10.294005+0000","last_became_peered":"2024-04-15T15:15:10.294005+0000","last_unstale":"2024-04-15T15:15:15.308185+0000","last_undegraded":"2024-04-15T15:15:15.308185+0000","last_fullsized":"2024-04-15T15:15:15.308185+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393556+0000","last_change":"2024-04-15T15:15:09.288397+0000","last_active":"2024-04-15T15:15:15.393556+0000","last_peered":"2024-04-15T15:15:15.393556+0000","last_clean":"2024-04-15T15:15:15.393556+0000","last_became_active":"2024-04-15T15:15:09.288135+0000","last_became_peered":"2024-04-15T15:15:09.288135+0000","last_unstale":"2024-04-15T15:15:15.393556+0000","last_undegraded":"2024-04-15T15:15:15.393556+0000","last_fullsized":"2024-04-15T15:15:15.393556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:05.360747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394110+0000","last_change":"2024-04-15T15:15:14.488972+0000","last_active":"2024-04-15T15:15:15.394110+0000","last_peered":"2024-04-15T15:15:15.394110+0000","last_clean":"2024-04-15T15:15:15.394110+0000","last_became_active":"2024-04-15T15:15:14.488302+0000","last_became_peered":"2024-04-15T15:15:14.488302+0000","last_unstale":"2024-04-15T15:15:15.394110+0000","last_undegraded":"2024-04-15T15:15:15.394110+0000","last_fullsized":"2024-04-15T15:15:15.394110+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310517+0000","last_change":"2024-04-15T15:15:10.294428+0000","last_active":"2024-04-15T15:15:15.310517+0000","last_peered":"2024-04-15T15:15:15.310517+0000","last_clean":"2024-04-15T15:15:15.310517+0000","last_became_active":"2024-04-15T15:15:10.294121+0000","last_became_peered":"2024-04-15T15:15:10.294121+0000","last_unstale":"2024-04-15T15:15:15.310517+0000","last_undegraded":"2024-04-15T15:15:15.310517+0000","last_fullsized":"2024-04-15T15:15:15.310517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:06.364679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865099+0000","last_change":"2024-04-15T15:15:09.294900+0000","last_active":"2024-04-15T15:15:15.865099+0000","last_peered":"2024-04-15T15:15:15.865099+0000","last_clean":"2024-04-15T15:15:15.865099+0000","last_became_active":"2024-04-15T15:15:09.294601+0000","last_became_peered":"2024-04-15T15:15:09.294601+0000","last_unstale":"2024-04-15T15:15:15.865099+0000","last_undegraded":"2024-04-15T15:15:15.865099+0000","last_fullsized":"2024-04-15T15:15:15.865099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309437+0000","last_change":"2024-04-15T15:15:14.328664+0000","last_active":"2024-04-15T15:15:15.309437+0000","last_peered":"2024-04-15T15:15:15.309437+0000","last_clean":"2024-04-15T15:15:15.309437+0000","last_became_active":"2024-04-15T15:15:14.327796+0000","last_became_peered":"2024-04-15T15:15:14.327796+0000","last_unstale":"2024-04-15T15:15:15.309437+0000","last_undegraded":"2024-04-15T15:15:15.309437+0000","last_fullsized":"2024-04-15T15:15:15.309437+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:28.892025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865447+0000","last_change":"2024-04-15T15:15:10.288747+0000","last_active":"2024-04-15T15:15:15.865447+0000","last_peered":"2024-04-15T15:15:15.865447+0000","last_clean":"2024-04-15T15:15:15.865447+0000","last_became_active":"2024-04-15T15:15:10.288465+0000","last_became_peered":"2024-04-15T15:15:10.288465+0000","last_unstale":"2024-04-15T15:15:15.865447+0000","last_undegraded":"2024-04-15T15:15:15.865447+0000","last_fullsized":"2024-04-15T15:15:15.865447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311197+0000","last_change":"2024-04-15T15:15:09.291999+0000","last_active":"2024-04-15T15:15:15.311197+0000","last_peered":"2024-04-15T15:15:15.311197+0000","last_clean":"2024-04-15T15:15:15.311197+0000","last_became_active":"2024-04-15T15:15:09.291744+0000","last_became_peered":"2024-04-15T15:15:09.291744+0000","last_unstale":"2024-04-15T15:15:15.311197+0000","last_undegraded":"2024-04-15T15:15:15.311197+0000","last_fullsized":"2024-04-15T15:15:15.311197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310618+0000","last_change":"2024-04-15T15:15:14.491129+0000","last_active":"2024-04-15T15:15:15.310618+0000","last_peered":"2024-04-15T15:15:15.310618+0000","last_clean":"2024-04-15T15:15:15.310618+0000","last_became_active":"2024-04-15T15:15:14.490619+0000","last_became_peered":"2024-04-15T15:15:14.490619+0000","last_unstale":"2024-04-15T15:15:15.310618+0000","last_undegraded":"2024-04-15T15:15:15.310618+0000","last_fullsized":"2024-04-15T15:15:15.310618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:46.083837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308841+0000","last_change":"2024-04-15T15:15:10.297516+0000","last_active":"2024-04-15T15:15:15.308841+0000","last_peered":"2024-04-15T15:15:15.308841+0000","last_clean":"2024-04-15T15:15:15.308841+0000","last_became_active":"2024-04-15T15:15:10.297282+0000","last_became_peered":"2024-04-15T15:15:10.297282+0000","last_unstale":"2024-04-15T15:15:15.308841+0000","last_undegraded":"2024-04-15T15:15:15.308841+0000","last_fullsized":"2024-04-15T15:15:15.308841+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394314+0000","last_change":"2024-04-15T15:15:09.288389+0000","last_active":"2024-04-15T15:15:15.394314+0000","last_peered":"2024-04-15T15:15:15.394314+0000","last_clean":"2024-04-15T15:15:15.394314+0000","last_became_active":"2024-04-15T15:15:09.288101+0000","last_became_peered":"2024-04-15T15:15:09.288101+0000","last_unstale":"2024-04-15T15:15:15.394314+0000","last_undegraded":"2024-04-15T15:15:15.394314+0000","last_fullsized":"2024-04-15T15:15:15.394314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:58.534133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.307926+0000","last_change":"2024-04-15T15:15:14.488661+0000","last_active":"2024-04-15T15:15:15.307926+0000","last_peered":"2024-04-15T15:15:15.307926+0000","last_clean":"2024-04-15T15:15:15.307926+0000","last_became_active":"2024-04-15T15:15:14.487944+0000","last_became_peered":"2024-04-15T15:15:14.487944+0000","last_unstale":"2024-04-15T15:15:15.307926+0000","last_undegraded":"2024-04-15T15:15:15.307926+0000","last_fullsized":"2024-04-15T15:15:15.307926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:36.475872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393929+0000","last_change":"2024-04-15T15:15:10.289924+0000","last_active":"2024-04-15T15:15:15.393929+0000","last_peered":"2024-04-15T15:15:15.393929+0000","last_clean":"2024-04-15T15:15:15.393929+0000","last_became_active":"2024-04-15T15:15:10.289662+0000","last_became_peered":"2024-04-15T15:15:10.289662+0000","last_unstale":"2024-04-15T15:15:15.393929+0000","last_undegraded":"2024-04-15T15:15:15.393929+0000","last_fullsized":"2024-04-15T15:15:15.393929+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866240+0000","last_change":"2024-04-15T15:15:09.295866+0000","last_active":"2024-04-15T15:15:15.866240+0000","last_peered":"2024-04-15T15:15:15.866240+0000","last_clean":"2024-04-15T15:15:15.866240+0000","last_became_active":"2024-04-15T15:15:09.295507+0000","last_became_peered":"2024-04-15T15:15:09.295507+0000","last_unstale":"2024-04-15T15:15:15.866240+0000","last_undegraded":"2024-04-15T15:15:15.866240+0000","last_fullsized":"2024-04-15T15:15:15.866240+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:09:01.918524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865520+0000","last_change":"2024-04-15T15:15:14.487233+0000","last_active":"2024-04-15T15:15:15.865520+0000","last_peered":"2024-04-15T15:15:15.865520+0000","last_clean":"2024-04-15T15:15:15.865520+0000","last_became_active":"2024-04-15T15:15:14.486846+0000","last_became_peered":"2024-04-15T15:15:14.486846+0000","last_unstale":"2024-04-15T15:15:15.865520+0000","last_undegraded":"2024-04-15T15:15:15.865520+0000","last_fullsized":"2024-04-15T15:15:15.865520+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:51.373166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865899+0000","last_change":"2024-04-15T15:15:10.299154+0000","last_active":"2024-04-15T15:15:15.865899+0000","last_peered":"2024-04-15T15:15:15.865899+0000","last_clean":"2024-04-15T15:15:15.865899+0000","last_became_active":"2024-04-15T15:15:10.298711+0000","last_became_peered":"2024-04-15T15:15:10.298711+0000","last_unstale":"2024-04-15T15:15:15.865899+0000","last_undegraded":"2024-04-15T15:15:15.865899+0000","last_fullsized":"2024-04-15T15:15:15.865899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394042+0000","last_change":"2024-04-15T15:15:09.289347+0000","last_active":"2024-04-15T15:15:15.394042+0000","last_peered":"2024-04-15T15:15:15.394042+0000","last_clean":"2024-04-15T15:15:15.394042+0000","last_became_active":"2024-04-15T15:15:09.289062+0000","last_became_peered":"2024-04-15T15:15:09.289062+0000","last_unstale":"2024-04-15T15:15:15.394042+0000","last_undegraded":"2024-04-15T15:15:15.394042+0000","last_fullsized":"2024-04-15T15:15:15.394042+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393588+0000","last_change":"2024-04-15T15:15:14.330539+0000","last_active":"2024-04-15T15:15:15.393588+0000","last_peered":"2024-04-15T15:15:15.393588+0000","last_clean":"2024-04-15T15:15:15.393588+0000","last_became_active":"2024-04-15T15:15:14.330173+0000","last_became_peered":"2024-04-15T15:15:14.330173+0000","last_unstale":"2024-04-15T15:15:15.393588+0000","last_undegraded":"2024-04-15T15:15:15.393588+0000","last_fullsized":"2024-04-15T15:15:15.393588+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310032+0000","last_change":"2024-04-15T15:15:10.286603+0000","last_active":"2024-04-15T15:15:15.310032+0000","last_peered":"2024-04-15T15:15:15.310032+0000","last_clean":"2024-04-15T15:15:15.310032+0000","last_became_active":"2024-04-15T15:15:10.286306+0000","last_became_peered":"2024-04-15T15:15:10.286306+0000","last_unstale":"2024-04-15T15:15:15.310032+0000","last_undegraded":"2024-04-15T15:15:15.310032+0000","last_fullsized":"2024-04-15T15:15:15.310032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:47.785526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865601+0000","last_change":"2024-04-15T15:15:09.291918+0000","last_active":"2024-04-15T15:15:15.865601+0000","last_peered":"2024-04-15T15:15:15.865601+0000","last_clean":"2024-04-15T15:15:15.865601+0000","last_became_active":"2024-04-15T15:15:09.291664+0000","last_became_peered":"2024-04-15T15:15:09.291664+0000","last_unstale":"2024-04-15T15:15:15.865601+0000","last_undegraded":"2024-04-15T15:15:15.865601+0000","last_fullsized":"2024-04-15T15:15:15.865601+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393983+0000","last_change":"2024-04-15T15:15:14.329421+0000","last_active":"2024-04-15T15:15:15.393983+0000","last_peered":"2024-04-15T15:15:15.393983+0000","last_clean":"2024-04-15T15:15:15.393983+0000","last_became_active":"2024-04-15T15:15:14.329120+0000","last_became_peered":"2024-04-15T15:15:14.329120+0000","last_unstale":"2024-04-15T15:15:15.393983+0000","last_undegraded":"2024-04-15T15:15:15.393983+0000","last_fullsized":"2024-04-15T15:15:15.393983+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393608+0000","last_change":"2024-04-15T15:15:10.290334+0000","last_active":"2024-04-15T15:15:15.393608+0000","last_peered":"2024-04-15T15:15:15.393608+0000","last_clean":"2024-04-15T15:15:15.393608+0000","last_became_active":"2024-04-15T15:15:10.290075+0000","last_became_peered":"2024-04-15T15:15:10.290075+0000","last_unstale":"2024-04-15T15:15:15.393608+0000","last_undegraded":"2024-04-15T15:15:15.393608+0000","last_fullsized":"2024-04-15T15:15:15.393608+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864908+0000","last_change":"2024-04-15T15:15:09.291564+0000","last_active":"2024-04-15T15:15:15.864908+0000","last_peered":"2024-04-15T15:15:15.864908+0000","last_clean":"2024-04-15T15:15:15.864908+0000","last_became_active":"2024-04-15T15:15:09.291265+0000","last_became_peered":"2024-04-15T15:15:09.291265+0000","last_unstale":"2024-04-15T15:15:15.864908+0000","last_undegraded":"2024-04-15T15:15:15.864908+0000","last_fullsized":"2024-04-15T15:15:15.864908+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394137+0000","last_change":"2024-04-15T15:15:14.329535+0000","last_active":"2024-04-15T15:15:15.394137+0000","last_peered":"2024-04-15T15:15:15.394137+0000","last_clean":"2024-04-15T15:15:15.394137+0000","last_became_active":"2024-04-15T15:15:14.329173+0000","last_became_peered":"2024-04-15T15:15:14.329173+0000","last_unstale":"2024-04-15T15:15:15.394137+0000","last_undegraded":"2024-04-15T15:15:15.394137+0000","last_fullsized":"2024-04-15T15:15:15.394137+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864715+0000","last_change":"2024-04-15T15:15:10.292048+0000","last_active":"2024-04-15T15:15:15.864715+0000","last_peered":"2024-04-15T15:15:15.864715+0000","last_clean":"2024-04-15T15:15:15.864715+0000","last_became_active":"2024-04-15T15:15:10.291773+0000","last_became_peered":"2024-04-15T15:15:10.291773+0000","last_unstale":"2024-04-15T15:15:15.864715+0000","last_undegraded":"2024-04-15T15:15:15.864715+0000","last_fullsized":"2024-04-15T15:15:15.864715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:12:45.215354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865214+0000","last_change":"2024-04-15T15:15:09.295426+0000","last_active":"2024-04-15T15:15:15.865214+0000","last_peered":"2024-04-15T15:15:15.865214+0000","last_clean":"2024-04-15T15:15:15.865214+0000","last_became_active":"2024-04-15T15:15:09.295096+0000","last_became_peered":"2024-04-15T15:15:09.295096+0000","last_unstale":"2024-04-15T15:15:15.865214+0000","last_undegraded":"2024-04-15T15:15:15.865214+0000","last_fullsized":"2024-04-15T15:15:15.865214+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865346+0000","last_change":"2024-04-15T15:15:14.487282+0000","last_active":"2024-04-15T15:15:15.865346+0000","last_peered":"2024-04-15T15:15:15.865346+0000","last_clean":"2024-04-15T15:15:15.865346+0000","last_became_active":"2024-04-15T15:15:14.486873+0000","last_became_peered":"2024-04-15T15:15:14.486873+0000","last_unstale":"2024-04-15T15:15:15.865346+0000","last_undegraded":"2024-04-15T15:15:15.865346+0000","last_fullsized":"2024-04-15T15:15:15.865346+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:54.916292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308149+0000","last_change":"2024-04-15T15:15:10.301516+0000","last_active":"2024-04-15T15:15:15.308149+0000","last_peered":"2024-04-15T15:15:15.308149+0000","last_clean":"2024-04-15T15:15:15.308149+0000","last_became_active":"2024-04-15T15:15:10.301272+0000","last_became_peered":"2024-04-15T15:15:10.301272+0000","last_unstale":"2024-04-15T15:15:15.308149+0000","last_undegraded":"2024-04-15T15:15:15.308149+0000","last_fullsized":"2024-04-15T15:15:15.308149+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865920+0000","last_change":"2024-04-15T15:15:09.292652+0000","last_active":"2024-04-15T15:15:15.865920+0000","last_peered":"2024-04-15T15:15:15.865920+0000","last_clean":"2024-04-15T15:15:15.865920+0000","last_became_active":"2024-04-15T15:15:09.292366+0000","last_became_peered":"2024-04-15T15:15:09.292366+0000","last_unstale":"2024-04-15T15:15:15.865920+0000","last_undegraded":"2024-04-15T15:15:15.865920+0000","last_fullsized":"2024-04-15T15:15:15.865920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865328+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:15:15.865328+0000","last_peered":"2024-04-15T15:15:15.865328+0000","last_clean":"2024-04-15T15:15:15.865328+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:15:15.865328+0000","last_undegraded":"2024-04-15T15:15:15.865328+0000","last_fullsized":"2024-04-15T15:15:15.865328+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310218+0000","last_change":"2024-04-15T15:15:14.488973+0000","last_active":"2024-04-15T15:15:15.310218+0000","last_peered":"2024-04-15T15:15:15.310218+0000","last_clean":"2024-04-15T15:15:15.310218+0000","last_became_active":"2024-04-15T15:15:14.488151+0000","last_became_peered":"2024-04-15T15:15:14.488151+0000","last_unstale":"2024-04-15T15:15:15.310218+0000","last_undegraded":"2024-04-15T15:15:15.310218+0000","last_fullsized":"2024-04-15T15:15:15.310218+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:38.441780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865992+0000","last_change":"2024-04-15T15:15:10.303487+0000","last_active":"2024-04-15T15:15:15.865992+0000","last_peered":"2024-04-15T15:15:15.865992+0000","last_clean":"2024-04-15T15:15:15.865992+0000","last_became_active":"2024-04-15T15:15:10.303222+0000","last_became_peered":"2024-04-15T15:15:10.303222+0000","last_unstale":"2024-04-15T15:15:15.865992+0000","last_undegraded":"2024-04-15T15:15:15.865992+0000","last_fullsized":"2024-04-15T15:15:15.865992+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:09.292763+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:09.292482+0000","last_became_peered":"2024-04-15T15:15:09.292482+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310239+0000","last_change":"2024-04-15T15:15:14.487139+0000","last_active":"2024-04-15T15:15:15.310239+0000","last_peered":"2024-04-15T15:15:15.310239+0000","last_clean":"2024-04-15T15:15:15.310239+0000","last_became_active":"2024-04-15T15:15:14.486746+0000","last_became_peered":"2024-04-15T15:15:14.486746+0000","last_unstale":"2024-04-15T15:15:15.310239+0000","last_undegraded":"2024-04-15T15:15:15.310239+0000","last_fullsized":"2024-04-15T15:15:15.310239+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865734+0000","last_change":"2024-04-15T15:15:10.298528+0000","last_active":"2024-04-15T15:15:15.865734+0000","last_peered":"2024-04-15T15:15:15.865734+0000","last_clean":"2024-04-15T15:15:15.865734+0000","last_became_active":"2024-04-15T15:15:10.298175+0000","last_became_peered":"2024-04-15T15:15:10.298175+0000","last_unstale":"2024-04-15T15:15:15.865734+0000","last_undegraded":"2024-04-15T15:15:15.865734+0000","last_fullsized":"2024-04-15T15:15:15.865734+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:51:34.416310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394078+0000","last_change":"2024-04-15T15:15:09.287274+0000","last_active":"2024-04-15T15:15:15.394078+0000","last_peered":"2024-04-15T15:15:15.394078+0000","last_clean":"2024-04-15T15:15:15.394078+0000","last_became_active":"2024-04-15T15:15:09.287022+0000","last_became_peered":"2024-04-15T15:15:09.287022+0000","last_unstale":"2024-04-15T15:15:15.394078+0000","last_undegraded":"2024-04-15T15:15:15.394078+0000","last_fullsized":"2024-04-15T15:15:15.394078+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:17.321981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865617+0000","last_change":"2024-04-15T15:15:14.327367+0000","last_active":"2024-04-15T15:15:15.865617+0000","last_peered":"2024-04-15T15:15:15.865617+0000","last_clean":"2024-04-15T15:15:15.865617+0000","last_became_active":"2024-04-15T15:15:14.327012+0000","last_became_peered":"2024-04-15T15:15:14.327012+0000","last_unstale":"2024-04-15T15:15:15.865617+0000","last_undegraded":"2024-04-15T15:15:15.865617+0000","last_fullsized":"2024-04-15T15:15:15.865617+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:17.757200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866362+0000","last_change":"2024-04-15T15:15:10.300664+0000","last_active":"2024-04-15T15:15:15.866362+0000","last_peered":"2024-04-15T15:15:15.866362+0000","last_clean":"2024-04-15T15:15:15.866362+0000","last_became_active":"2024-04-15T15:15:10.300411+0000","last_became_peered":"2024-04-15T15:15:10.300411+0000","last_unstale":"2024-04-15T15:15:15.866362+0000","last_undegraded":"2024-04-15T15:15:15.866362+0000","last_fullsized":"2024-04-15T15:15:15.866362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:44:32.450402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864760+0000","last_change":"2024-04-15T15:15:09.291588+0000","last_active":"2024-04-15T15:15:15.864760+0000","last_peered":"2024-04-15T15:15:15.864760+0000","last_clean":"2024-04-15T15:15:15.864760+0000","last_became_active":"2024-04-15T15:15:09.291317+0000","last_became_peered":"2024-04-15T15:15:09.291317+0000","last_unstale":"2024-04-15T15:15:15.864760+0000","last_undegraded":"2024-04-15T15:15:15.864760+0000","last_fullsized":"2024-04-15T15:15:15.864760+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:37:08.623620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864847+0000","last_change":"2024-04-15T15:15:14.488073+0000","last_active":"2024-04-15T15:15:15.864847+0000","last_peered":"2024-04-15T15:15:15.864847+0000","last_clean":"2024-04-15T15:15:15.864847+0000","last_became_active":"2024-04-15T15:15:14.487630+0000","last_became_peered":"2024-04-15T15:15:14.487630+0000","last_unstale":"2024-04-15T15:15:15.864847+0000","last_undegraded":"2024-04-15T15:15:15.864847+0000","last_fullsized":"2024-04-15T15:15:15.864847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:20:38.196606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866072+0000","last_change":"2024-04-15T15:15:10.293789+0000","last_active":"2024-04-15T15:15:15.866072+0000","last_peered":"2024-04-15T15:15:15.866072+0000","last_clean":"2024-04-15T15:15:15.866072+0000","last_became_active":"2024-04-15T15:15:10.293508+0000","last_became_peered":"2024-04-15T15:15:10.293508+0000","last_unstale":"2024-04-15T15:15:15.866072+0000","last_undegraded":"2024-04-15T15:15:15.866072+0000","last_fullsized":"2024-04-15T15:15:15.866072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:13.731730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310273+0000","last_change":"2024-04-15T15:15:09.280627+0000","last_active":"2024-04-15T15:15:15.310273+0000","last_peered":"2024-04-15T15:15:15.310273+0000","last_clean":"2024-04-15T15:15:15.310273+0000","last_became_active":"2024-04-15T15:15:09.280327+0000","last_became_peered":"2024-04-15T15:15:09.280327+0000","last_unstale":"2024-04-15T15:15:15.310273+0000","last_undegraded":"2024-04-15T15:15:15.310273+0000","last_fullsized":"2024-04-15T15:15:15.310273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:15:24.741048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308985+0000","last_change":"2024-04-15T15:15:14.327503+0000","last_active":"2024-04-15T15:15:15.308985+0000","last_peered":"2024-04-15T15:15:15.308985+0000","last_clean":"2024-04-15T15:15:15.308985+0000","last_became_active":"2024-04-15T15:15:14.327045+0000","last_became_peered":"2024-04-15T15:15:14.327045+0000","last_unstale":"2024-04-15T15:15:15.308985+0000","last_undegraded":"2024-04-15T15:15:15.308985+0000","last_fullsized":"2024-04-15T15:15:15.308985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:41.707546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865253+0000","last_change":"2024-04-15T15:15:10.294511+0000","last_active":"2024-04-15T15:15:15.865253+0000","last_peered":"2024-04-15T15:15:15.865253+0000","last_clean":"2024-04-15T15:15:15.865253+0000","last_became_active":"2024-04-15T15:15:10.294217+0000","last_became_peered":"2024-04-15T15:15:10.294217+0000","last_unstale":"2024-04-15T15:15:15.865253+0000","last_undegraded":"2024-04-15T15:15:15.865253+0000","last_fullsized":"2024-04-15T15:15:15.865253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864350+0000","last_change":"2024-04-15T15:15:09.289902+0000","last_active":"2024-04-15T15:15:15.864350+0000","last_peered":"2024-04-15T15:15:15.864350+0000","last_clean":"2024-04-15T15:15:15.864350+0000","last_became_active":"2024-04-15T15:15:09.289521+0000","last_became_peered":"2024-04-15T15:15:09.289521+0000","last_unstale":"2024-04-15T15:15:15.864350+0000","last_undegraded":"2024-04-15T15:15:15.864350+0000","last_fullsized":"2024-04-15T15:15:15.864350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865201+0000","last_change":"2024-04-15T15:15:14.325169+0000","last_active":"2024-04-15T15:15:15.865201+0000","last_peered":"2024-04-15T15:15:15.865201+0000","last_clean":"2024-04-15T15:15:15.865201+0000","last_became_active":"2024-04-15T15:15:14.324680+0000","last_became_peered":"2024-04-15T15:15:14.324680+0000","last_unstale":"2024-04-15T15:15:15.865201+0000","last_undegraded":"2024-04-15T15:15:15.865201+0000","last_fullsized":"2024-04-15T15:15:15.865201+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308937+0000","last_change":"2024-04-15T15:15:10.291452+0000","last_active":"2024-04-15T15:15:15.308937+0000","last_peered":"2024-04-15T15:15:15.308937+0000","last_clean":"2024-04-15T15:15:15.308937+0000","last_became_active":"2024-04-15T15:15:10.291170+0000","last_became_peered":"2024-04-15T15:15:10.291170+0000","last_unstale":"2024-04-15T15:15:15.308937+0000","last_undegraded":"2024-04-15T15:15:15.308937+0000","last_fullsized":"2024-04-15T15:15:15.308937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309951+0000","last_change":"2024-04-15T15:15:09.292208+0000","last_active":"2024-04-15T15:15:15.309951+0000","last_peered":"2024-04-15T15:15:15.309951+0000","last_clean":"2024-04-15T15:15:15.309951+0000","last_became_active":"2024-04-15T15:15:09.290702+0000","last_became_peered":"2024-04-15T15:15:09.290702+0000","last_unstale":"2024-04-15T15:15:15.309951+0000","last_undegraded":"2024-04-15T15:15:15.309951+0000","last_fullsized":"2024-04-15T15:15:15.309951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308793+0000","last_change":"2024-04-15T15:15:14.488657+0000","last_active":"2024-04-15T15:15:15.308793+0000","last_peered":"2024-04-15T15:15:15.308793+0000","last_clean":"2024-04-15T15:15:15.308793+0000","last_became_active":"2024-04-15T15:15:14.488282+0000","last_became_peered":"2024-04-15T15:15:14.488282+0000","last_unstale":"2024-04-15T15:15:15.308793+0000","last_undegraded":"2024-04-15T15:15:15.308793+0000","last_fullsized":"2024-04-15T15:15:15.308793+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:21:03.495639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865696+0000","last_change":"2024-04-15T15:15:10.301823+0000","last_active":"2024-04-15T15:15:15.865696+0000","last_peered":"2024-04-15T15:15:15.865696+0000","last_clean":"2024-04-15T15:15:15.865696+0000","last_became_active":"2024-04-15T15:15:10.301545+0000","last_became_peered":"2024-04-15T15:15:10.301545+0000","last_unstale":"2024-04-15T15:15:15.865696+0000","last_undegraded":"2024-04-15T15:15:15.865696+0000","last_fullsized":"2024-04-15T15:15:15.865696+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865481+0000","last_change":"2024-04-15T15:15:09.291724+0000","last_active":"2024-04-15T15:15:15.865481+0000","last_peered":"2024-04-15T15:15:15.865481+0000","last_clean":"2024-04-15T15:15:15.865481+0000","last_became_active":"2024-04-15T15:15:09.291390+0000","last_became_peered":"2024-04-15T15:15:09.291390+0000","last_unstale":"2024-04-15T15:15:15.865481+0000","last_undegraded":"2024-04-15T15:15:15.865481+0000","last_fullsized":"2024-04-15T15:15:15.865481+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308524+0000","last_change":"2024-04-15T15:15:14.491287+0000","last_active":"2024-04-15T15:15:15.308524+0000","last_peered":"2024-04-15T15:15:15.308524+0000","last_clean":"2024-04-15T15:15:15.308524+0000","last_became_active":"2024-04-15T15:15:14.488643+0000","last_became_peered":"2024-04-15T15:15:14.488643+0000","last_unstale":"2024-04-15T15:15:15.308524+0000","last_undegraded":"2024-04-15T15:15:15.308524+0000","last_fullsized":"2024-04-15T15:15:15.308524+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:29:47.252030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865184+0000","last_change":"2024-04-15T15:15:10.293687+0000","last_active":"2024-04-15T15:15:15.865184+0000","last_peered":"2024-04-15T15:15:15.865184+0000","last_clean":"2024-04-15T15:15:15.865184+0000","last_became_active":"2024-04-15T15:15:10.293430+0000","last_became_peered":"2024-04-15T15:15:10.293430+0000","last_unstale":"2024-04-15T15:15:15.865184+0000","last_undegraded":"2024-04-15T15:15:15.865184+0000","last_fullsized":"2024-04-15T15:15:15.865184+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865653+0000","last_change":"2024-04-15T15:15:09.282027+0000","last_active":"2024-04-15T15:15:15.865653+0000","last_peered":"2024-04-15T15:15:15.865653+0000","last_clean":"2024-04-15T15:15:15.865653+0000","last_became_active":"2024-04-15T15:15:09.281730+0000","last_became_peered":"2024-04-15T15:15:09.281730+0000","last_unstale":"2024-04-15T15:15:15.865653+0000","last_undegraded":"2024-04-15T15:15:15.865653+0000","last_fullsized":"2024-04-15T15:15:15.865653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:32:01.954710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866059+0000","last_change":"2024-04-15T15:15:14.489189+0000","last_active":"2024-04-15T15:15:15.866059+0000","last_peered":"2024-04-15T15:15:15.866059+0000","last_clean":"2024-04-15T15:15:15.866059+0000","last_became_active":"2024-04-15T15:15:14.488758+0000","last_became_peered":"2024-04-15T15:15:14.488758+0000","last_unstale":"2024-04-15T15:15:15.866059+0000","last_undegraded":"2024-04-15T15:15:15.866059+0000","last_fullsized":"2024-04-15T15:15:15.866059+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865898+0000","last_change":"2024-04-15T15:15:10.298047+0000","last_active":"2024-04-15T15:15:15.865898+0000","last_peered":"2024-04-15T15:15:15.865898+0000","last_clean":"2024-04-15T15:15:15.865898+0000","last_became_active":"2024-04-15T15:15:10.297779+0000","last_became_peered":"2024-04-15T15:15:10.297779+0000","last_unstale":"2024-04-15T15:15:15.865898+0000","last_undegraded":"2024-04-15T15:15:15.865898+0000","last_fullsized":"2024-04-15T15:15:15.865898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":31,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:19.789685+0000","last_change":"2024-04-15T15:15:09.290413+0000","last_active":"2024-04-15T15:15:19.789685+0000","last_peered":"2024-04-15T15:15:19.789685+0000","last_clean":"2024-04-15T15:15:19.789685+0000","last_became_active":"2024-04-15T15:15:09.289971+0000","last_became_peered":"2024-04-15T15:15:09.289971+0000","last_unstale":"2024-04-15T15:15:19.789685+0000","last_undegraded":"2024-04-15T15:15:19.789685+0000","last_fullsized":"2024-04-15T15:15:19.789685+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393701+0000","last_change":"2024-04-15T15:15:14.324351+0000","last_active":"2024-04-15T15:15:15.393701+0000","last_peered":"2024-04-15T15:15:15.393701+0000","last_clean":"2024-04-15T15:15:15.393701+0000","last_became_active":"2024-04-15T15:15:14.322177+0000","last_became_peered":"2024-04-15T15:15:14.322177+0000","last_unstale":"2024-04-15T15:15:15.393701+0000","last_undegraded":"2024-04-15T15:15:15.393701+0000","last_fullsized":"2024-04-15T15:15:15.393701+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310092+0000","last_change":"2024-04-15T15:15:10.290888+0000","last_active":"2024-04-15T15:15:15.310092+0000","last_peered":"2024-04-15T15:15:15.310092+0000","last_clean":"2024-04-15T15:15:15.310092+0000","last_became_active":"2024-04-15T15:15:10.290630+0000","last_became_peered":"2024-04-15T15:15:10.290630+0000","last_unstale":"2024-04-15T15:15:15.310092+0000","last_undegraded":"2024-04-15T15:15:15.310092+0000","last_fullsized":"2024-04-15T15:15:15.310092+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394037+0000","last_change":"2024-04-15T15:15:09.291950+0000","last_active":"2024-04-15T15:15:15.394037+0000","last_peered":"2024-04-15T15:15:15.394037+0000","last_clean":"2024-04-15T15:15:15.394037+0000","last_became_active":"2024-04-15T15:15:09.291687+0000","last_became_peered":"2024-04-15T15:15:09.291687+0000","last_unstale":"2024-04-15T15:15:15.394037+0000","last_undegraded":"2024-04-15T15:15:15.394037+0000","last_fullsized":"2024-04-15T15:15:15.394037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309744+0000","last_change":"2024-04-15T15:15:14.328229+0000","last_active":"2024-04-15T15:15:15.309744+0000","last_peered":"2024-04-15T15:15:15.309744+0000","last_clean":"2024-04-15T15:15:15.309744+0000","last_became_active":"2024-04-15T15:15:14.327167+0000","last_became_peered":"2024-04-15T15:15:14.327167+0000","last_unstale":"2024-04-15T15:15:15.309744+0000","last_undegraded":"2024-04-15T15:15:15.309744+0000","last_fullsized":"2024-04-15T15:15:15.309744+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:36.054042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865433+0000","last_change":"2024-04-15T15:15:10.300821+0000","last_active":"2024-04-15T15:15:15.865433+0000","last_peered":"2024-04-15T15:15:15.865433+0000","last_clean":"2024-04-15T15:15:15.865433+0000","last_became_active":"2024-04-15T15:15:10.300561+0000","last_became_peered":"2024-04-15T15:15:10.300561+0000","last_unstale":"2024-04-15T15:15:15.865433+0000","last_undegraded":"2024-04-15T15:15:15.865433+0000","last_fullsized":"2024-04-15T15:15:15.865433+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865315+0000","last_change":"2024-04-15T15:15:09.294709+0000","last_active":"2024-04-15T15:15:15.865315+0000","last_peered":"2024-04-15T15:15:15.865315+0000","last_clean":"2024-04-15T15:15:15.865315+0000","last_became_active":"2024-04-15T15:15:09.294135+0000","last_became_peered":"2024-04-15T15:15:09.294135+0000","last_unstale":"2024-04-15T15:15:15.865315+0000","last_undegraded":"2024-04-15T15:15:15.865315+0000","last_fullsized":"2024-04-15T15:15:15.865315+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308127+0000","last_change":"2024-04-15T15:15:14.332302+0000","last_active":"2024-04-15T15:15:15.308127+0000","last_peered":"2024-04-15T15:15:15.308127+0000","last_clean":"2024-04-15T15:15:15.308127+0000","last_became_active":"2024-04-15T15:15:14.331893+0000","last_became_peered":"2024-04-15T15:15:14.331893+0000","last_unstale":"2024-04-15T15:15:15.308127+0000","last_undegraded":"2024-04-15T15:15:15.308127+0000","last_fullsized":"2024-04-15T15:15:15.308127+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:18:57.970154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865270+0000","last_change":"2024-04-15T15:15:10.295887+0000","last_active":"2024-04-15T15:15:15.865270+0000","last_peered":"2024-04-15T15:15:15.865270+0000","last_clean":"2024-04-15T15:15:15.865270+0000","last_became_active":"2024-04-15T15:15:10.295591+0000","last_became_peered":"2024-04-15T15:15:10.295591+0000","last_unstale":"2024-04-15T15:15:15.865270+0000","last_undegraded":"2024-04-15T15:15:15.865270+0000","last_fullsized":"2024-04-15T15:15:15.865270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311117+0000","last_change":"2024-04-15T15:15:09.290489+0000","last_active":"2024-04-15T15:15:15.311117+0000","last_peered":"2024-04-15T15:15:15.311117+0000","last_clean":"2024-04-15T15:15:15.311117+0000","last_became_active":"2024-04-15T15:15:09.290207+0000","last_became_peered":"2024-04-15T15:15:09.290207+0000","last_unstale":"2024-04-15T15:15:15.311117+0000","last_undegraded":"2024-04-15T15:15:15.311117+0000","last_fullsized":"2024-04-15T15:15:15.311117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:29.760839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308129+0000","last_change":"2024-04-15T15:15:14.490596+0000","last_active":"2024-04-15T15:15:15.308129+0000","last_peered":"2024-04-15T15:15:15.308129+0000","last_clean":"2024-04-15T15:15:15.308129+0000","last_became_active":"2024-04-15T15:15:14.489609+0000","last_became_peered":"2024-04-15T15:15:14.489609+0000","last_unstale":"2024-04-15T15:15:15.308129+0000","last_undegraded":"2024-04-15T15:15:15.308129+0000","last_fullsized":"2024-04-15T15:15:15.308129+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:04:06.401428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865063+0000","last_change":"2024-04-15T15:15:14.489631+0000","last_active":"2024-04-15T15:15:15.865063+0000","last_peered":"2024-04-15T15:15:15.865063+0000","last_clean":"2024-04-15T15:15:15.865063+0000","last_became_active":"2024-04-15T15:15:14.489248+0000","last_became_peered":"2024-04-15T15:15:14.489248+0000","last_unstale":"2024-04-15T15:15:15.865063+0000","last_undegraded":"2024-04-15T15:15:15.865063+0000","last_fullsized":"2024-04-15T15:15:15.865063+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:57.684992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393445+0000","last_change":"2024-04-15T15:15:10.293423+0000","last_active":"2024-04-15T15:15:15.393445+0000","last_peered":"2024-04-15T15:15:15.393445+0000","last_clean":"2024-04-15T15:15:15.393445+0000","last_became_active":"2024-04-15T15:15:10.293101+0000","last_became_peered":"2024-04-15T15:15:10.293101+0000","last_unstale":"2024-04-15T15:15:15.393445+0000","last_undegraded":"2024-04-15T15:15:15.393445+0000","last_fullsized":"2024-04-15T15:15:15.393445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865159+0000","last_change":"2024-04-15T15:15:09.290248+0000","last_active":"2024-04-15T15:15:15.865159+0000","last_peered":"2024-04-15T15:15:15.865159+0000","last_clean":"2024-04-15T15:15:15.865159+0000","last_became_active":"2024-04-15T15:15:09.289792+0000","last_became_peered":"2024-04-15T15:15:09.289792+0000","last_unstale":"2024-04-15T15:15:15.865159+0000","last_undegraded":"2024-04-15T15:15:15.865159+0000","last_fullsized":"2024-04-15T15:15:15.865159+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:18:46.781594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.311080+0000","last_change":"2024-04-15T15:15:14.323489+0000","last_active":"2024-04-15T15:15:15.311080+0000","last_peered":"2024-04-15T15:15:15.311080+0000","last_clean":"2024-04-15T15:15:15.311080+0000","last_became_active":"2024-04-15T15:15:14.322956+0000","last_became_peered":"2024-04-15T15:15:14.322956+0000","last_unstale":"2024-04-15T15:15:15.311080+0000","last_undegraded":"2024-04-15T15:15:15.311080+0000","last_fullsized":"2024-04-15T15:15:15.311080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:17.400117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308115+0000","last_change":"2024-04-15T15:15:10.295446+0000","last_active":"2024-04-15T15:15:15.308115+0000","last_peered":"2024-04-15T15:15:15.308115+0000","last_clean":"2024-04-15T15:15:15.308115+0000","last_became_active":"2024-04-15T15:15:10.295165+0000","last_became_peered":"2024-04-15T15:15:10.295165+0000","last_unstale":"2024-04-15T15:15:15.308115+0000","last_undegraded":"2024-04-15T15:15:15.308115+0000","last_fullsized":"2024-04-15T15:15:15.308115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308300+0000","last_change":"2024-04-15T15:15:09.289277+0000","last_active":"2024-04-15T15:15:15.308300+0000","last_peered":"2024-04-15T15:15:15.308300+0000","last_clean":"2024-04-15T15:15:15.308300+0000","last_became_active":"2024-04-15T15:15:09.289026+0000","last_became_peered":"2024-04-15T15:15:09.289026+0000","last_unstale":"2024-04-15T15:15:15.308300+0000","last_undegraded":"2024-04-15T15:15:15.308300+0000","last_fullsized":"2024-04-15T15:15:15.308300+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:23:41.236636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310503+0000","last_change":"2024-04-15T15:15:09.285504+0000","last_active":"2024-04-15T15:15:15.310503+0000","last_peered":"2024-04-15T15:15:15.310503+0000","last_clean":"2024-04-15T15:15:15.310503+0000","last_became_active":"2024-04-15T15:15:09.285219+0000","last_became_peered":"2024-04-15T15:15:09.285219+0000","last_unstale":"2024-04-15T15:15:15.310503+0000","last_undegraded":"2024-04-15T15:15:15.310503+0000","last_fullsized":"2024-04-15T15:15:15.310503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:05:28.705962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865651+0000","last_change":"2024-04-15T15:15:10.297720+0000","last_active":"2024-04-15T15:15:15.865651+0000","last_peered":"2024-04-15T15:15:15.865651+0000","last_clean":"2024-04-15T15:15:15.865651+0000","last_became_active":"2024-04-15T15:15:10.297469+0000","last_became_peered":"2024-04-15T15:15:10.297469+0000","last_unstale":"2024-04-15T15:15:15.865651+0000","last_undegraded":"2024-04-15T15:15:15.865651+0000","last_fullsized":"2024-04-15T15:15:15.865651+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864649+0000","last_change":"2024-04-15T15:15:14.327342+0000","last_active":"2024-04-15T15:15:15.864649+0000","last_peered":"2024-04-15T15:15:15.864649+0000","last_clean":"2024-04-15T15:15:15.864649+0000","last_became_active":"2024-04-15T15:15:14.326929+0000","last_became_peered":"2024-04-15T15:15:14.326929+0000","last_unstale":"2024-04-15T15:15:15.864649+0000","last_undegraded":"2024-04-15T15:15:15.864649+0000","last_fullsized":"2024-04-15T15:15:15.864649+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:19:56.492223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308298+0000","last_change":"2024-04-15T15:15:09.288401+0000","last_active":"2024-04-15T15:15:15.308298+0000","last_peered":"2024-04-15T15:15:15.308298+0000","last_clean":"2024-04-15T15:15:15.308298+0000","last_became_active":"2024-04-15T15:15:09.288124+0000","last_became_peered":"2024-04-15T15:15:09.288124+0000","last_unstale":"2024-04-15T15:15:15.308298+0000","last_undegraded":"2024-04-15T15:15:15.308298+0000","last_fullsized":"2024-04-15T15:15:15.308298+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:08.266537+0000","last_clean_scrub_stamp":"2024-04-15T15:15:08.266537+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:17:53.369222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309493+0000","last_change":"2024-04-15T15:15:10.293787+0000","last_active":"2024-04-15T15:15:15.309493+0000","last_peered":"2024-04-15T15:15:15.309493+0000","last_clean":"2024-04-15T15:15:15.309493+0000","last_became_active":"2024-04-15T15:15:10.293409+0000","last_became_peered":"2024-04-15T15:15:10.293409+0000","last_unstale":"2024-04-15T15:15:15.309493+0000","last_undegraded":"2024-04-15T15:15:15.309493+0000","last_fullsized":"2024-04-15T15:15:15.309493+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:30.764771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308821+0000","last_change":"2024-04-15T15:15:14.323853+0000","last_active":"2024-04-15T15:15:15.308821+0000","last_peered":"2024-04-15T15:15:15.308821+0000","last_clean":"2024-04-15T15:15:15.308821+0000","last_became_active":"2024-04-15T15:15:14.323547+0000","last_became_peered":"2024-04-15T15:15:14.323547+0000","last_unstale":"2024-04-15T15:15:15.308821+0000","last_undegraded":"2024-04-15T15:15:15.308821+0000","last_fullsized":"2024-04-15T15:15:15.308821+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:13.295939+0000","last_clean_scrub_stamp":"2024-04-15T15:15:13.295939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:58:48.237751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308815+0000","last_change":"2024-04-15T15:15:10.290641+0000","last_active":"2024-04-15T15:15:15.308815+0000","last_peered":"2024-04-15T15:15:15.308815+0000","last_clean":"2024-04-15T15:15:15.308815+0000","last_became_active":"2024-04-15T15:15:10.290342+0000","last_became_peered":"2024-04-15T15:15:10.290342+0000","last_unstale":"2024-04-15T15:15:15.308815+0000","last_undegraded":"2024-04-15T15:15:15.308815+0000","last_fullsized":"2024-04-15T15:15:15.308815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:09:02.416980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864807+0000","last_change":"2024-04-15T15:15:10.302121+0000","last_active":"2024-04-15T15:15:15.864807+0000","last_peered":"2024-04-15T15:15:15.864807+0000","last_clean":"2024-04-15T15:15:15.864807+0000","last_became_active":"2024-04-15T15:15:10.301860+0000","last_became_peered":"2024-04-15T15:15:10.301860+0000","last_unstale":"2024-04-15T15:15:15.864807+0000","last_undegraded":"2024-04-15T15:15:15.864807+0000","last_fullsized":"2024-04-15T15:15:15.864807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:54:59.538065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865207+0000","last_change":"2024-04-15T15:15:10.298052+0000","last_active":"2024-04-15T15:15:15.865207+0000","last_peered":"2024-04-15T15:15:15.865207+0000","last_clean":"2024-04-15T15:15:15.865207+0000","last_became_active":"2024-04-15T15:15:10.296656+0000","last_became_peered":"2024-04-15T15:15:10.296656+0000","last_unstale":"2024-04-15T15:15:15.865207+0000","last_undegraded":"2024-04-15T15:15:15.865207+0000","last_fullsized":"2024-04-15T15:15:15.865207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865604+0000","last_change":"2024-04-15T15:15:10.296027+0000","last_active":"2024-04-15T15:15:15.865604+0000","last_peered":"2024-04-15T15:15:15.865604+0000","last_clean":"2024-04-15T15:15:15.865604+0000","last_became_active":"2024-04-15T15:15:10.295741+0000","last_became_peered":"2024-04-15T15:15:10.295741+0000","last_unstale":"2024-04-15T15:15:15.865604+0000","last_undegraded":"2024-04-15T15:15:15.865604+0000","last_fullsized":"2024-04-15T15:15:15.865604+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308554+0000","last_change":"2024-04-15T15:15:10.287343+0000","last_active":"2024-04-15T15:15:15.308554+0000","last_peered":"2024-04-15T15:15:15.308554+0000","last_clean":"2024-04-15T15:15:15.308554+0000","last_became_active":"2024-04-15T15:15:10.287082+0000","last_became_peered":"2024-04-15T15:15:10.287082+0000","last_unstale":"2024-04-15T15:15:15.308554+0000","last_undegraded":"2024-04-15T15:15:15.308554+0000","last_fullsized":"2024-04-15T15:15:15.308554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:58:41.893816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393974+0000","last_change":"2024-04-15T15:15:10.289020+0000","last_active":"2024-04-15T15:15:15.393974+0000","last_peered":"2024-04-15T15:15:15.393974+0000","last_clean":"2024-04-15T15:15:15.393974+0000","last_became_active":"2024-04-15T15:15:10.288777+0000","last_became_peered":"2024-04-15T15:15:10.288777+0000","last_unstale":"2024-04-15T15:15:15.393974+0000","last_undegraded":"2024-04-15T15:15:15.393974+0000","last_fullsized":"2024-04-15T15:15:15.393974+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310906+0000","last_change":"2024-04-15T15:15:10.290594+0000","last_active":"2024-04-15T15:15:15.310906+0000","last_peered":"2024-04-15T15:15:15.310906+0000","last_clean":"2024-04-15T15:15:15.310906+0000","last_became_active":"2024-04-15T15:15:10.290345+0000","last_became_peered":"2024-04-15T15:15:10.290345+0000","last_unstale":"2024-04-15T15:15:15.310906+0000","last_undegraded":"2024-04-15T15:15:15.310906+0000","last_fullsized":"2024-04-15T15:15:15.310906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308747+0000","last_change":"2024-04-15T15:15:10.297500+0000","last_active":"2024-04-15T15:15:15.308747+0000","last_peered":"2024-04-15T15:15:15.308747+0000","last_clean":"2024-04-15T15:15:15.308747+0000","last_became_active":"2024-04-15T15:15:10.296730+0000","last_became_peered":"2024-04-15T15:15:10.296730+0000","last_unstale":"2024-04-15T15:15:15.308747+0000","last_undegraded":"2024-04-15T15:15:15.308747+0000","last_fullsized":"2024-04-15T15:15:15.308747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:22:24.866555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865052+0000","last_change":"2024-04-15T15:15:10.296225+0000","last_active":"2024-04-15T15:15:15.865052+0000","last_peered":"2024-04-15T15:15:15.865052+0000","last_clean":"2024-04-15T15:15:15.865052+0000","last_became_active":"2024-04-15T15:15:10.295930+0000","last_became_peered":"2024-04-15T15:15:10.295930+0000","last_unstale":"2024-04-15T15:15:15.865052+0000","last_undegraded":"2024-04-15T15:15:15.865052+0000","last_fullsized":"2024-04-15T15:15:15.865052+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308196+0000","last_change":"2024-04-15T15:15:10.289427+0000","last_active":"2024-04-15T15:15:15.308196+0000","last_peered":"2024-04-15T15:15:15.308196+0000","last_clean":"2024-04-15T15:15:15.308196+0000","last_became_active":"2024-04-15T15:15:10.289170+0000","last_became_peered":"2024-04-15T15:15:10.289170+0000","last_unstale":"2024-04-15T15:15:15.308196+0000","last_undegraded":"2024-04-15T15:15:15.308196+0000","last_fullsized":"2024-04-15T15:15:15.308196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:10:21.039029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865103+0000","last_change":"2024-04-15T15:15:10.304028+0000","last_active":"2024-04-15T15:15:15.865103+0000","last_peered":"2024-04-15T15:15:15.865103+0000","last_clean":"2024-04-15T15:15:15.865103+0000","last_became_active":"2024-04-15T15:15:10.303771+0000","last_became_peered":"2024-04-15T15:15:10.303771+0000","last_unstale":"2024-04-15T15:15:15.865103+0000","last_undegraded":"2024-04-15T15:15:15.865103+0000","last_fullsized":"2024-04-15T15:15:15.865103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:13.374647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865533+0000","last_change":"2024-04-15T15:15:10.293639+0000","last_active":"2024-04-15T15:15:15.865533+0000","last_peered":"2024-04-15T15:15:15.865533+0000","last_clean":"2024-04-15T15:15:15.865533+0000","last_became_active":"2024-04-15T15:15:10.293220+0000","last_became_peered":"2024-04-15T15:15:10.293220+0000","last_unstale":"2024-04-15T15:15:15.865533+0000","last_undegraded":"2024-04-15T15:15:15.865533+0000","last_fullsized":"2024-04-15T15:15:15.865533+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310554+0000","last_change":"2024-04-15T15:15:10.289725+0000","last_active":"2024-04-15T15:15:15.310554+0000","last_peered":"2024-04-15T15:15:15.310554+0000","last_clean":"2024-04-15T15:15:15.310554+0000","last_became_active":"2024-04-15T15:15:10.289378+0000","last_became_peered":"2024-04-15T15:15:10.289378+0000","last_unstale":"2024-04-15T15:15:15.310554+0000","last_undegraded":"2024-04-15T15:15:15.310554+0000","last_fullsized":"2024-04-15T15:15:15.310554+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866173+0000","last_change":"2024-04-15T15:15:10.295231+0000","last_active":"2024-04-15T15:15:15.866173+0000","last_peered":"2024-04-15T15:15:15.866173+0000","last_clean":"2024-04-15T15:15:15.866173+0000","last_became_active":"2024-04-15T15:15:10.294974+0000","last_became_peered":"2024-04-15T15:15:10.294974+0000","last_unstale":"2024-04-15T15:15:15.866173+0000","last_undegraded":"2024-04-15T15:15:15.866173+0000","last_fullsized":"2024-04-15T15:15:15.866173+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:41:47.347696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864735+0000","last_change":"2024-04-15T15:15:10.300143+0000","last_active":"2024-04-15T15:15:15.864735+0000","last_peered":"2024-04-15T15:15:15.864735+0000","last_clean":"2024-04-15T15:15:15.864735+0000","last_became_active":"2024-04-15T15:15:10.299889+0000","last_became_peered":"2024-04-15T15:15:10.299889+0000","last_unstale":"2024-04-15T15:15:15.864735+0000","last_undegraded":"2024-04-15T15:15:15.864735+0000","last_fullsized":"2024-04-15T15:15:15.864735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:16:46.735175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865763+0000","last_change":"2024-04-15T15:15:10.295205+0000","last_active":"2024-04-15T15:15:15.865763+0000","last_peered":"2024-04-15T15:15:15.865763+0000","last_clean":"2024-04-15T15:15:15.865763+0000","last_became_active":"2024-04-15T15:15:10.294556+0000","last_became_peered":"2024-04-15T15:15:10.294556+0000","last_unstale":"2024-04-15T15:15:15.865763+0000","last_undegraded":"2024-04-15T15:15:15.865763+0000","last_fullsized":"2024-04-15T15:15:15.865763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864921+0000","last_change":"2024-04-15T15:15:10.297025+0000","last_active":"2024-04-15T15:15:15.864921+0000","last_peered":"2024-04-15T15:15:15.864921+0000","last_clean":"2024-04-15T15:15:15.864921+0000","last_became_active":"2024-04-15T15:15:10.296519+0000","last_became_peered":"2024-04-15T15:15:10.296519+0000","last_unstale":"2024-04-15T15:15:15.864921+0000","last_undegraded":"2024-04-15T15:15:15.864921+0000","last_fullsized":"2024-04-15T15:15:15.864921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.393530+0000","last_change":"2024-04-15T15:15:10.288655+0000","last_active":"2024-04-15T15:15:15.393530+0000","last_peered":"2024-04-15T15:15:15.393530+0000","last_clean":"2024-04-15T15:15:15.393530+0000","last_became_active":"2024-04-15T15:15:10.286513+0000","last_became_peered":"2024-04-15T15:15:10.286513+0000","last_unstale":"2024-04-15T15:15:15.393530+0000","last_undegraded":"2024-04-15T15:15:15.393530+0000","last_fullsized":"2024-04-15T15:15:15.393530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865152+0000","last_change":"2024-04-15T15:15:10.299829+0000","last_active":"2024-04-15T15:15:15.865152+0000","last_peered":"2024-04-15T15:15:15.865152+0000","last_clean":"2024-04-15T15:15:15.865152+0000","last_became_active":"2024-04-15T15:15:10.299281+0000","last_became_peered":"2024-04-15T15:15:10.299281+0000","last_unstale":"2024-04-15T15:15:15.865152+0000","last_undegraded":"2024-04-15T15:15:15.865152+0000","last_fullsized":"2024-04-15T15:15:15.865152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:26:12.807802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308094+0000","last_change":"2024-04-15T15:15:10.301135+0000","last_active":"2024-04-15T15:15:15.308094+0000","last_peered":"2024-04-15T15:15:15.308094+0000","last_clean":"2024-04-15T15:15:15.308094+0000","last_became_active":"2024-04-15T15:15:10.300869+0000","last_became_peered":"2024-04-15T15:15:10.300869+0000","last_unstale":"2024-04-15T15:15:15.308094+0000","last_undegraded":"2024-04-15T15:15:15.308094+0000","last_fullsized":"2024-04-15T15:15:15.308094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:07:32.574280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.309542+0000","last_change":"2024-04-15T15:15:10.285169+0000","last_active":"2024-04-15T15:15:15.309542+0000","last_peered":"2024-04-15T15:15:15.309542+0000","last_clean":"2024-04-15T15:15:15.309542+0000","last_became_active":"2024-04-15T15:15:10.284906+0000","last_became_peered":"2024-04-15T15:15:10.284906+0000","last_unstale":"2024-04-15T15:15:15.309542+0000","last_undegraded":"2024-04-15T15:15:15.309542+0000","last_fullsized":"2024-04-15T15:15:15.309542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:24:18.325913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310364+0000","last_change":"2024-04-15T15:15:10.300827+0000","last_active":"2024-04-15T15:15:15.310364+0000","last_peered":"2024-04-15T15:15:15.310364+0000","last_clean":"2024-04-15T15:15:15.310364+0000","last_became_active":"2024-04-15T15:15:10.300530+0000","last_became_peered":"2024-04-15T15:15:10.300530+0000","last_unstale":"2024-04-15T15:15:15.310364+0000","last_undegraded":"2024-04-15T15:15:15.310364+0000","last_fullsized":"2024-04-15T15:15:15.310364+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.310951+0000","last_change":"2024-04-15T15:15:10.297517+0000","last_active":"2024-04-15T15:15:15.310951+0000","last_peered":"2024-04-15T15:15:15.310951+0000","last_clean":"2024-04-15T15:15:15.310951+0000","last_became_active":"2024-04-15T15:15:10.297224+0000","last_became_peered":"2024-04-15T15:15:10.297224+0000","last_unstale":"2024-04-15T15:15:15.310951+0000","last_undegraded":"2024-04-15T15:15:15.310951+0000","last_fullsized":"2024-04-15T15:15:15.310951+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:41:37.682076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866349+0000","last_change":"2024-04-15T15:15:10.305356+0000","last_active":"2024-04-15T15:15:15.866349+0000","last_peered":"2024-04-15T15:15:15.866349+0000","last_clean":"2024-04-15T15:15:15.866349+0000","last_became_active":"2024-04-15T15:15:10.305035+0000","last_became_peered":"2024-04-15T15:15:10.305035+0000","last_unstale":"2024-04-15T15:15:15.866349+0000","last_undegraded":"2024-04-15T15:15:15.866349+0000","last_fullsized":"2024-04-15T15:15:15.866349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:00:23.860901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.308001+0000","last_change":"2024-04-15T15:15:10.299528+0000","last_active":"2024-04-15T15:15:15.308001+0000","last_peered":"2024-04-15T15:15:15.308001+0000","last_clean":"2024-04-15T15:15:15.308001+0000","last_became_active":"2024-04-15T15:15:10.299272+0000","last_became_peered":"2024-04-15T15:15:10.299272+0000","last_unstale":"2024-04-15T15:15:15.308001+0000","last_undegraded":"2024-04-15T15:15:15.308001+0000","last_fullsized":"2024-04-15T15:15:15.308001+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:16:46.279895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865187+0000","last_change":"2024-04-15T15:15:10.295288+0000","last_active":"2024-04-15T15:15:15.865187+0000","last_peered":"2024-04-15T15:15:15.865187+0000","last_clean":"2024-04-15T15:15:15.865187+0000","last_became_active":"2024-04-15T15:15:10.295000+0000","last_became_peered":"2024-04-15T15:15:10.295000+0000","last_unstale":"2024-04-15T15:15:15.865187+0000","last_undegraded":"2024-04-15T15:15:15.865187+0000","last_fullsized":"2024-04-15T15:15:15.865187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:02:32.890094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.864972+0000","last_change":"2024-04-15T15:15:10.298860+0000","last_active":"2024-04-15T15:15:15.864972+0000","last_peered":"2024-04-15T15:15:15.864972+0000","last_clean":"2024-04-15T15:15:15.864972+0000","last_became_active":"2024-04-15T15:15:10.296935+0000","last_became_peered":"2024-04-15T15:15:10.296935+0000","last_unstale":"2024-04-15T15:15:15.864972+0000","last_undegraded":"2024-04-15T15:15:15.864972+0000","last_fullsized":"2024-04-15T15:15:15.864972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:24:53.659522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865016+0000","last_change":"2024-04-15T15:15:10.296047+0000","last_active":"2024-04-15T15:15:15.865016+0000","last_peered":"2024-04-15T15:15:15.865016+0000","last_clean":"2024-04-15T15:15:15.865016+0000","last_became_active":"2024-04-15T15:15:10.295590+0000","last_became_peered":"2024-04-15T15:15:10.295590+0000","last_unstale":"2024-04-15T15:15:15.865016+0000","last_undegraded":"2024-04-15T15:15:15.865016+0000","last_fullsized":"2024-04-15T15:15:15.865016+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:56:42.058367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865557+0000","last_change":"2024-04-15T15:15:10.298993+0000","last_active":"2024-04-15T15:15:15.865557+0000","last_peered":"2024-04-15T15:15:15.865557+0000","last_clean":"2024-04-15T15:15:15.865557+0000","last_became_active":"2024-04-15T15:15:10.298652+0000","last_became_peered":"2024-04-15T15:15:10.298652+0000","last_unstale":"2024-04-15T15:15:15.865557+0000","last_undegraded":"2024-04-15T15:15:15.865557+0000","last_fullsized":"2024-04-15T15:15:15.865557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:20:32.028572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.865100+0000","last_change":"2024-04-15T15:15:10.291981+0000","last_active":"2024-04-15T15:15:15.865100+0000","last_peered":"2024-04-15T15:15:15.865100+0000","last_clean":"2024-04-15T15:15:15.865100+0000","last_became_active":"2024-04-15T15:15:10.291495+0000","last_became_peered":"2024-04-15T15:15:10.291495+0000","last_unstale":"2024-04-15T15:15:15.865100+0000","last_undegraded":"2024-04-15T15:15:15.865100+0000","last_fullsized":"2024-04-15T15:15:15.865100+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:41:36.018411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.394232+0000","last_change":"2024-04-15T15:15:10.288087+0000","last_active":"2024-04-15T15:15:15.394232+0000","last_peered":"2024-04-15T15:15:15.394232+0000","last_clean":"2024-04-15T15:15:15.394232+0000","last_became_active":"2024-04-15T15:15:10.287803+0000","last_became_peered":"2024-04-15T15:15:10.287803+0000","last_unstale":"2024-04-15T15:15:15.394232+0000","last_undegraded":"2024-04-15T15:15:15.394232+0000","last_fullsized":"2024-04-15T15:15:15.394232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:10:52.508168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-15T15:15:15.866222+0000","last_change":"2024-04-15T15:15:10.302148+0000","last_active":"2024-04-15T15:15:15.866222+0000","last_peered":"2024-04-15T15:15:15.866222+0000","last_clean":"2024-04-15T15:15:15.866222+0000","last_became_active":"2024-04-15T15:15:10.301882+0000","last_became_peered":"2024-04-15T15:15:10.301882+0000","last_unstale":"2024-04-15T15:15:15.866222+0000","last_undegraded":"2024-04-15T15:15:15.866222+0000","last_fullsized":"2024-04-15T15:15:15.866222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:15:09.270469+0000","last_clean_scrub_stamp":"2024-04-15T15:15:09.270469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:51:50.890822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343976,"statfs":{"total":96636764160,"available":96608231424,"internally_reserved":0,"allocated":864256,"data_stored":695597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":278528,"data_stored":108489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":104737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":104773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574857,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":104647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":104807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":872448,"data_stored":695653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:15:32.124 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:15:32.124 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T15:15:32.124 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-15T15:15:32.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-15T15:15:32.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.258+0000 7fc2315ba640 1 Processor -- start 2024-04-15T15:15:32.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.258+0000 7fc2315ba640 1 -- start start 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc2315ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc22c09a9d0 0x7fc22c09adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc2315ba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc22c06cd00 0x7fc22c162ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc2315ba640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc22c15ec70 con 0x7fc22c09b720 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc2315ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc22c15f840 con 0x7fc22c09a9d0 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc2315ba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc22c09b2f0 con 0x7fc22c06cd00 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc22affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc22c09a9d0 0x7fc22c09adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc22a7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc22c09b720 0x7fc22c1623d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44818/0 (socket says 172.21.15.70:44818) 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.259+0000 7fc22a7fc640 1 -- 172.21.15.70:0/3021555058 learned_addr learned my addr 172.21.15.70:0/3021555058 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22b7fe640 1 --2- 172.21.15.70:0/3021555058 >> v2:172.21.15.96:3300/0 conn(0x7fc22c06cd00 0x7fc22c162ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22affd640 1 -- 172.21.15.70:0/3021555058 >> v2:172.21.15.96:3300/0 conn(0x7fc22c06cd00 msgr2=0x7fc22c162ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22affd640 1 --2- 172.21.15.70:0/3021555058 >> v2:172.21.15.96:3300/0 conn(0x7fc22c06cd00 0x7fc22c162ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22affd640 1 -- 172.21.15.70:0/3021555058 >> v1:172.21.15.70:6789/0 conn(0x7fc22c09b720 legacy=0x7fc22c1623d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22affd640 1 -- 172.21.15.70:0/3021555058 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c0b1580 con 0x7fc22c09a9d0 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc229ffb640 1 -- 172.21.15.70:0/3021555058 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535422559 0 0) 0x7fc22c15ec70 con 0x7fc22c09b720 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc22affd640 1 --2- 172.21.15.70:0/3021555058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09a9d0 0x7fc22c09adb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc22c163470 tx=0x7fc22402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd319f859516648 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc229ffb640 1 -- 172.21.15.70:0/3021555058 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224039070 con 0x7fc22c09a9d0 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.260+0000 7fc229ffb640 1 -- 172.21.15.70:0/3021555058 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc22402cc80 con 0x7fc22c09a9d0 2024-04-15T15:15:32.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc229ffb640 1 -- 172.21.15.70:0/3021555058 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc224034660 con 0x7fc22c09a9d0 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 -- 172.21.15.70:0/3021555058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09a9d0 msgr2=0x7fc22c09adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 --2- 172.21.15.70:0/3021555058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09a9d0 0x7fc22c09adb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc22c163470 tx=0x7fc22402bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 -- 172.21.15.70:0/3021555058 shutdown_connections 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 --2- 172.21.15.70:0/3021555058 >> v2:172.21.15.96:3300/0 conn(0x7fc22c06cd00 0x7fc22c162ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 --2- 172.21.15.70:0/3021555058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09a9d0 0x7fc22c09adb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 -- 172.21.15.70:0/3021555058 >> 172.21.15.70:0/3021555058 conn(0x7fc22c0b3e70 msgr2=0x7fc22c0b4280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 -- 172.21.15.70:0/3021555058 shutdown_connections 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.261+0000 7fc2315ba640 1 -- 172.21.15.70:0/3021555058 wait complete. 2024-04-15T15:15:32.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.262+0000 7fc2315ba640 1 Processor -- start 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.262+0000 7fc2315ba640 1 -- start start 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 0x7fc22c160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 0x7fc22c0b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c15f1f0 con 0x7fc22c09b720 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c15ef10 con 0x7fc22c09a9d0 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc2315ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc22c15f4d0 con 0x7fc22c06cd00 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:36900/0 (socket says 172.21.15.70:36900) 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 -- 172.21.15.70:0/2234321786 learned_addr learned my addr 172.21.15.70:0/2234321786 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22affd640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 0x7fc22c160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22a7fc640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 0x7fc22c0b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 -- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 msgr2=0x7fc22c160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 0x7fc22c160d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 -- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 msgr2=0x7fc22c0b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 0x7fc22c0b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22b7fe640 1 -- 172.21.15.70:0/2234321786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c161300 con 0x7fc22c09b720 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22affd640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 0x7fc22c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.263+0000 7fc22a7fc640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 0x7fc22c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc22b7fe640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c550 tx=0x7fc21c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c015070 con 0x7fc22c09b720 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc21c004220 con 0x7fc22c09b720 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c1615e0 con 0x7fc22c09b720 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c003610 con 0x7fc22c09b720 2024-04-15T15:15:32.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.264+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c161ba0 con 0x7fc22c09b720 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.265+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc22c0690f0 con 0x7fc22c09b720 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.265+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc21c0037b0 con 0x7fc22c09b720 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.266+0000 7fc2237fe640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.266+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc21c08bbb0 con 0x7fc22c09b720 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.266+0000 7fc22affd640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.266+0000 7fc22affd640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 0x7fc20c06ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc224036000 tx=0x7fc224030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.270+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc21c059d30 con 0x7fc22c09b720 2024-04-15T15:15:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.484+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc22c072610 con 0x7fc22c09b720 2024-04-15T15:15:32.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.484+0000 7fc2237fe640 1 -- 172.21.15.70:0/2234321786 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc21c059630 con 0x7fc22c09b720 2024-04-15T15:15:32.486 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:32.486 INFO:teuthology.orchestra.run.smithi070.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 msgr2=0x7fc20c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 0x7fc20c06ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc224036000 tx=0x7fc224030000 comp rx=0 tx=0).stop 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 msgr2=0x7fc22c0b50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c550 tx=0x7fc21c00ca10 comp rx=0 tx=0).stop 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 shutdown_connections 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc20c06c920 0x7fc20c06ede0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc22c09b720 0x7fc22c0b50f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc22c09a9d0 0x7fc22c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 --2- 172.21.15.70:0/2234321786 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc22c06cd00 0x7fc22c160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 >> 172.21.15.70:0/2234321786 conn(0x7fc22c0b3e70 msgr2=0x7fc22c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 shutdown_connections 2024-04-15T15:15:32.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.486+0000 7fc2315ba640 1 -- 172.21.15.70:0/2234321786 wait complete. 2024-04-15T15:15:32.499 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-15T15:15:32.499 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:32.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.600+0000 7f116e39d640 1 Processor -- start 2024-04-15T15:15:32.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.600+0000 7f116e39d640 1 -- start start 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116e39d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11680cf210 0x7f11680d2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116e39d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 0x7f11680d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116e39d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11680d1e60 con 0x7f116815e9a0 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116e39d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f11680d15d0 con 0x7f11680cf210 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116e39d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f11680d18a0 con 0x7f11680cf690 2024-04-15T15:15:32.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11680cf210 0x7f11680d2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f11680cf210 0x7f11680d2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50410/0 (socket says 172.21.15.70:50410) 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 -- 172.21.15.70:0/1705026776 learned_addr learned my addr 172.21.15.70:0/1705026776 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116c93a640 1 --2- 172.21.15.70:0/1705026776 >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 0x7f11680d30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 -- 172.21.15.70:0/1705026776 >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 msgr2=0x7f11680d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 --2- 172.21.15.70:0/1705026776 >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 0x7f11680d30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 -- 172.21.15.70:0/1705026776 >> v1:172.21.15.96:6789/0 conn(0x7f116815e9a0 legacy=0x7f11680d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f115ffff640 1 -- 172.21.15.70:0/1705026776 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11680d1b80 con 0x7f11680cf210 2024-04-15T15:15:32.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.601+0000 7f116c93a640 1 --2- 172.21.15.70:0/1705026776 >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 0x7f11680d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:32.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.602+0000 7f115ffff640 1 --2- 172.21.15.70:0/1705026776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11680cf210 0x7f11680d2ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f116806c7a0 tx=0x7f115002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930e8f7e75a59874 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.602+0000 7f11677fe640 1 -- 172.21.15.70:0/1705026776 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150035020 con 0x7f11680cf210 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.602+0000 7f11677fe640 1 -- 172.21.15.70:0/1705026776 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1150031870 con 0x7f11680cf210 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.602+0000 7f11677fe640 1 -- 172.21.15.70:0/1705026776 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1150031c40 con 0x7f11680cf210 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 -- 172.21.15.70:0/1705026776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11680cf210 msgr2=0x7f11680d2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 --2- 172.21.15.70:0/1705026776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11680cf210 0x7f11680d2ba0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f116806c7a0 tx=0x7f115002b910 comp rx=0 tx=0).stop 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 -- 172.21.15.70:0/1705026776 shutdown_connections 2024-04-15T15:15:32.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 --2- 172.21.15.70:0/1705026776 >> v2:172.21.15.96:3300/0 conn(0x7f11680cf690 0x7f11680d30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 --2- 172.21.15.70:0/1705026776 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11680cf210 0x7f11680d2ba0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 -- 172.21.15.70:0/1705026776 >> 172.21.15.70:0/1705026776 conn(0x7f116809d950 msgr2=0x7f116808bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 -- 172.21.15.70:0/1705026776 shutdown_connections 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.603+0000 7f116e39d640 1 -- 172.21.15.70:0/1705026776 wait complete. 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.604+0000 7f116e39d640 1 Processor -- start 2024-04-15T15:15:32.607 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.604+0000 7f116e39d640 1 -- start start 2024-04-15T15:15:32.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11680cf210 0x7f1168153b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 0x7f11680c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11680d2360 con 0x7f116815e9a0 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f116806d2d0 con 0x7f11680cf690 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f11680d24e0 con 0x7f11680cf210 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50414/0 (socket says 172.21.15.70:50414) 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 -- 172.21.15.70:0/1830008775 learned_addr learned my addr 172.21.15.70:0/1830008775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f115ffff640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 0x7f11680c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f1167fff640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11680cf210 0x7f1168153b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 -- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11680cf210 msgr2=0x7f1168153b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11680cf210 0x7f1168153b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 -- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 msgr2=0x7f11680c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 0x7f11680c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 -- 172.21.15.70:0/1830008775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11681604f0 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f115ffff640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 0x7f11680c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116c93a640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f116000cef0 tx=0x7f116000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116001a070 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116000d860 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11680c8f20 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116000dc50 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.605+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11680c94e0 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.606+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11680690f0 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.607+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f116001d070 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.607+0000 7f11657fa640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 0x7f114806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.608+0000 7f1167fff640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 0x7f114806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.608+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f116008c760 con 0x7f116815e9a0 2024-04-15T15:15:32.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.608+0000 7f1167fff640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 0x7f114806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1158004ea0 tx=0x7f1158009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.612+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f116005a910 con 0x7f116815e9a0 2024-04-15T15:15:32.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.787+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f116806cd30 con 0x7f116815e9a0 2024-04-15T15:15:32.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.788+0000 7f11657fa640 1 -- 172.21.15.70:0/1830008775 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f116005a210 con 0x7f116815e9a0 2024-04-15T15:15:32.790 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:32.790 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 msgr2=0x7f114806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 0x7f114806edb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1158004ea0 tx=0x7f1158009040 comp rx=0 tx=0).stop 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 msgr2=0x7f11680ca020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f116000cef0 tx=0x7f116000a710 comp rx=0 tx=0).stop 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 shutdown_connections 2024-04-15T15:15:32.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f114806c8f0 0x7f114806edb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f116815e9a0 0x7f11680ca020 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f11680cf690 0x7f11680c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 --2- 172.21.15.70:0/1830008775 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f11680cf210 0x7f1168153b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 >> 172.21.15.70:0/1830008775 conn(0x7f116809d950 msgr2=0x7f116806d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 shutdown_connections 2024-04-15T15:15:32.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.790+0000 7f116e39d640 1 -- 172.21.15.70:0/1830008775 wait complete. 2024-04-15T15:15:32.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:32.804 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.908+0000 7f04515e9640 1 Processor -- start 2024-04-15T15:15:32.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.908+0000 7f04515e9640 1 -- start start 2024-04-15T15:15:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f04515e9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f044c0cd000 0x7f044c0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f04515e9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f044c09ab40 0x7f044c09afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f04515e9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f044c15dbd0 con 0x7f044c1602d0 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f04515e9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f044c0cd460 con 0x7f044c09ab40 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f04515e9640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f044c0cd6a0 con 0x7f044c0cd000 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f044c09ab40 0x7f044c09afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f044c09ab40 0x7f044c09afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50422/0 (socket says 172.21.15.70:50422) 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 -- 172.21.15.70:0/1843717550 learned_addr learned my addr 172.21.15.70:0/1843717550 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044a7fc640 1 --2- 172.21.15.70:0/1843717550 >> v2:172.21.15.96:3300/0 conn(0x7f044c0cd000 0x7f044c0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 -- 172.21.15.70:0/1843717550 >> v2:172.21.15.96:3300/0 conn(0x7f044c0cd000 msgr2=0x7f044c0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 --2- 172.21.15.70:0/1843717550 >> v2:172.21.15.96:3300/0 conn(0x7f044c0cd000 0x7f044c0d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 -- 172.21.15.70:0/1843717550 >> v1:172.21.15.96:6789/0 conn(0x7f044c1602d0 legacy=0x7f044c1606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.909+0000 7f044b7fe640 1 -- 172.21.15.70:0/1843717550 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c0cd980 con 0x7f044c09ab40 2024-04-15T15:15:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.910+0000 7f044b7fe640 1 --2- 172.21.15.70:0/1843717550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c09ab40 0x7f044c09afd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f044c0a05a0 tx=0x7f044402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92168f01c5e321c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.910+0000 7f0449ffb640 1 -- 172.21.15.70:0/1843717550 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444034070 con 0x7f044c09ab40 2024-04-15T15:15:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.910+0000 7f0449ffb640 1 -- 172.21.15.70:0/1843717550 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444030860 con 0x7f044c09ab40 2024-04-15T15:15:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.910+0000 7f0449ffb640 1 -- 172.21.15.70:0/1843717550 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444030c30 con 0x7f044c09ab40 2024-04-15T15:15:32.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 -- 172.21.15.70:0/1843717550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c09ab40 msgr2=0x7f044c09afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 --2- 172.21.15.70:0/1843717550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c09ab40 0x7f044c09afd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f044c0a05a0 tx=0x7f044402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 -- 172.21.15.70:0/1843717550 shutdown_connections 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 --2- 172.21.15.70:0/1843717550 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c09ab40 0x7f044c09afd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 --2- 172.21.15.70:0/1843717550 >> v2:172.21.15.96:3300/0 conn(0x7f044c0cd000 0x7f044c0d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 -- 172.21.15.70:0/1843717550 >> 172.21.15.70:0/1843717550 conn(0x7f044c0d2460 msgr2=0x7f044c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 -- 172.21.15.70:0/1843717550 shutdown_connections 2024-04-15T15:15:32.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.911+0000 7f04515e9640 1 -- 172.21.15.70:0/1843717550 wait complete. 2024-04-15T15:15:32.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.912+0000 7f04515e9640 1 Processor -- start 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.912+0000 7f04515e9640 1 -- start start 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f044c09ab40 0x7f044c0b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f044c0cd000 0x7f044c0b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f044c0ce220 con 0x7f044c1602d0 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f044c0cdf40 con 0x7f044c09ab40 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f04515e9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f044c0d1430 con 0x7f044c0cd000 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50438/0 (socket says 172.21.15.70:50438) 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 -- 172.21.15.70:0/1042022527 learned_addr learned my addr 172.21.15.70:0/1042022527 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044affd640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f044c09ab40 0x7f044c0b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044a7fc640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f044c0cd000 0x7f044c0b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 -- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f044c0cd000 msgr2=0x7f044c0b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f044c0cd000 0x7f044c0b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 -- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f044c09ab40 msgr2=0x7f044c0b02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f044c09ab40 0x7f044c0b02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 -- 172.21.15.70:0/1042022527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f044c15ad10 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.913+0000 7f044b7fe640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f044402b9e0 tx=0x7f04440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.914+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444034040 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.914+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0444039d90 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.914+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0444043630 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.914+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f044c1f3c30 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.914+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f044c1f69c0 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.915+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f044c0690f0 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.915+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0444037070 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.916+0000 7f04337fe640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.916+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f04440ade80 con 0x7f044c1602d0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.916+0000 7f044affd640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:32.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.916+0000 7f044affd640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 0x7f043806edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f044c155540 tx=0x7f043c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:32.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:32.920+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f044c0690f0 con 0x7f044c1602d0 2024-04-15T15:15:33.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.093+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f044c0d16b0 con 0x7f044c1602d0 2024-04-15T15:15:33.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.093+0000 7f04337fe640 1 -- 172.21.15.70:0/1042022527 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f044c0d16b0 con 0x7f044c1602d0 2024-04-15T15:15:33.095 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 msgr2=0x7f043806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 0x7f043806edb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f044c155540 tx=0x7f043c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 msgr2=0x7f044c158b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f044402b9e0 tx=0x7f04440046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 shutdown_connections 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f043806c8f0 0x7f043806edb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f044c1602d0 0x7f044c158b10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f044c0cd000 0x7f044c0b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 --2- 172.21.15.70:0/1042022527 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f044c09ab40 0x7f044c0b02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 >> 172.21.15.70:0/1042022527 conn(0x7f044c0d2460 msgr2=0x7f044c1606b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 shutdown_connections 2024-04-15T15:15:33.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.095+0000 7f04515e9640 1 -- 172.21.15.70:0/1042022527 wait complete. 2024-04-15T15:15:33.108 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:33.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.212+0000 7fd44c814640 1 Processor -- start 2024-04-15T15:15:33.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.212+0000 7fd44c814640 1 -- start start 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.213+0000 7fd44c814640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd448153670 0x7fd4480d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.213+0000 7fd44c814640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44806b9e0 con 0x7fd4480d0dc0 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.213+0000 7fd44c814640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd44806bc80 con 0x7fd4480d21e0 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.213+0000 7fd44c814640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd44806bf60 con 0x7fd448153670 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd446d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd4480d0dc0 0x7fd4480cf160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43248/0 (socket says 172.21.15.70:43248) 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd446d9d640 1 -- 172.21.15.70:0/3444536822 learned_addr learned my addr 172.21.15.70:0/3444536822 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44659c640 1 --2- 172.21.15.70:0/3444536822 >> v2:172.21.15.96:3300/0 conn(0x7fd448153670 0x7fd4480d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44559a640 1 -- 172.21.15.70:0/3444536822 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571685372 0 0) 0x7fd44806b9e0 con 0x7fd4480d0dc0 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44559a640 1 -- 172.21.15.70:0/3444536822 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd434003520 con 0x7fd4480d0dc0 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44659c640 1 -- 172.21.15.70:0/3444536822 >> v1:172.21.15.96:6789/0 conn(0x7fd4480d21e0 legacy=0x7fd4480d06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44659c640 1 -- 172.21.15.70:0/3444536822 >> v1:172.21.15.70:6789/0 conn(0x7fd4480d0dc0 legacy=0x7fd4480cf160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.214+0000 7fd44659c640 1 -- 172.21.15.70:0/3444536822 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44806c240 con 0x7fd448153670 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44659c640 1 --2- 172.21.15.70:0/3444536822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd4480d1ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd438009d90 tx=0x7fd43802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1783fe17572dd311 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44559a640 1 -- 172.21.15.70:0/3444536822 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438039070 con 0x7fd448153670 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44559a640 1 -- 172.21.15.70:0/3444536822 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd43802cc60 con 0x7fd448153670 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44559a640 1 -- 172.21.15.70:0/3444536822 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438034640 con 0x7fd448153670 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44c814640 1 -- 172.21.15.70:0/3444536822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 msgr2=0x7fd4480d1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.215+0000 7fd44c814640 1 --2- 172.21.15.70:0/3444536822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd4480d1ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd438009d90 tx=0x7fd43802beb0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.216+0000 7fd44c814640 1 -- 172.21.15.70:0/3444536822 shutdown_connections 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.216+0000 7fd44c814640 1 --2- 172.21.15.70:0/3444536822 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd4480d1ca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.216+0000 7fd44c814640 1 -- 172.21.15.70:0/3444536822 >> 172.21.15.70:0/3444536822 conn(0x7fd44809a9d0 msgr2=0x7fd4480d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.216+0000 7fd44c814640 1 -- 172.21.15.70:0/3444536822 shutdown_connections 2024-04-15T15:15:33.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.216+0000 7fd44c814640 1 -- 172.21.15.70:0/3444536822 wait complete. 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.217+0000 7fd44c814640 1 Processor -- start 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.217+0000 7fd44c814640 1 -- start start 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd4480d0dc0 0x7fd4480c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd44806ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd44815db40 con 0x7fd4480d21e0 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd44815d9c0 con 0x7fd448153670 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd4480cf870 con 0x7fd4480d0dc0 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50452/0 (socket says 172.21.15.70:50452) 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 -- 172.21.15.70:0/4055109003 learned_addr learned my addr 172.21.15.70:0/4055109003 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd446d9d640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd44806ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44659c640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd4480d0dc0 0x7fd4480c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 -- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd4480d0dc0 msgr2=0x7fd4480c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd4480d0dc0 0x7fd4480c4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 -- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 msgr2=0x7fd44806ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd44806ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 -- 172.21.15.70:0/4055109003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44806f930 con 0x7fd4480d21e0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd445d9b640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c430 tx=0x7fd43000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4300052d0 con 0x7fd4480d21e0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd430005490 con 0x7fd4480d21e0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd430007260 con 0x7fd4480d21e0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44806fc10 con 0x7fd4480d21e0 2024-04-15T15:15:33.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.218+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd448155390 con 0x7fd4480d21e0 2024-04-15T15:15:33.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.220+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd448155590 con 0x7fd4480d21e0 2024-04-15T15:15:33.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.220+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd430007400 con 0x7fd4480d21e0 2024-04-15T15:15:33.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.220+0000 7fd43effd640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 0x7fd42806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.220+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd43008b3f0 con 0x7fd4480d21e0 2024-04-15T15:15:33.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.220+0000 7fd44659c640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 0x7fd42806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.221+0000 7fd44659c640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 0x7fd42806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd438036000 tx=0x7fd438030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.225+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd448155590 con 0x7fd4480d21e0 2024-04-15T15:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.402+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd44814cff0 con 0x7fd4480d21e0 2024-04-15T15:15:33.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.402+0000 7fd43effd640 1 -- 172.21.15.70:0/4055109003 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fd4300594e0 con 0x7fd4480d21e0 2024-04-15T15:15:33.404 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:33.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 msgr2=0x7fd42806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 0x7fd42806edb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd438036000 tx=0x7fd438030000 comp rx=0 tx=0).stop 2024-04-15T15:15:33.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 msgr2=0x7fd44814ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd43000c430 tx=0x7fd43000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 shutdown_connections 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd448153670 0x7fd44806ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd42806c8f0 0x7fd42806edb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd4480d21e0 0x7fd44814ca50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 --2- 172.21.15.70:0/4055109003 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd4480d0dc0 0x7fd4480c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 >> 172.21.15.70:0/4055109003 conn(0x7fd44809a9d0 msgr2=0x7fd4480b0030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 shutdown_connections 2024-04-15T15:15:33.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.404+0000 7fd44c814640 1 -- 172.21.15.70:0/4055109003 wait complete. 2024-04-15T15:15:33.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:33.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.522+0000 7f3fd2c69640 1 Processor -- start 2024-04-15T15:15:33.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.522+0000 7f3fd2c69640 1 -- start start 2024-04-15T15:15:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fd2c69640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fcc167600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fd2c69640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fcc15dbd0 con 0x7f3fcc0cd000 2024-04-15T15:15:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fd2c69640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fcc0cd6a0 con 0x7f3fcc1602d0 2024-04-15T15:15:33.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fd2c69640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc0cd460 con 0x7f3fcc09bb50 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3fcc0cd000 0x7f3fcc0d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43260/0 (socket says 172.21.15.70:43260) 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc3fff640 1 -- 172.21.15.70:0/1538184094 learned_addr learned my addr 172.21.15.70:0/1538184094 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fd1206640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fcc167600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1840714935 0 0) 0x7f3fcc15dbd0 con 0x7f3fcc0cd000 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3fb4003520 con 0x7f3fcc0cd000 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3424266017 0 0) 0x7f3fb4003520 con 0x7f3fcc0cd000 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 msgr2=0x7f3fcc167600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fcc167600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 >> v1:172.21.15.96:6789/0 conn(0x7f3fcc1602d0 legacy=0x7f3fcc1606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc0cd980 con 0x7f3fcc0cd000 2024-04-15T15:15:33.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.523+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f3fbc001670 con 0x7f3fcc0cd000 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 >> v1:172.21.15.70:6789/0 conn(0x7f3fcc0cd000 legacy=0x7f3fcc0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb40039e0 0x7f3fb4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fb40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0cd460 con 0x7f3fb40039e0 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fb400baa0 con 0x7f3fb4004800 2024-04-15T15:15:33.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3fb400bca0 con 0x7f3fcc09bb50 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb40039e0 0x7f3fb4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fc3fff640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd1206640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fb40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 -- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 msgr2=0x7f3fb40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fb40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 -- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fb4004800 msgr2=0x7f3fb4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fb4004800 0x7f3fb4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd0a05640 1 -- 172.21.15.70:0/1538184094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fb4003520 con 0x7f3fb40039e0 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.524+0000 7f3fd1206640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fcc09bb50 0x7f3fb40051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:33.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb40039e0 0x7f3fb4003e00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8001720 tx=0x7f3fb4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1674992e28176ba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8001370 con 0x7f3fb40039e0 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc80052d0 con 0x7f3fb40039e0 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fc37fe640 1 -- 172.21.15.70:0/1538184094 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc8003830 con 0x7f3fb40039e0 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1538184094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb40039e0 msgr2=0x7f3fb4003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1538184094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fb40039e0 0x7f3fb4003e00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8001720 tx=0x7f3fb4005830 comp rx=0 tx=0).stop 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd1206640 1 -- 172.21.15.70:0/1538184094 reap_dead start 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1538184094 shutdown_connections 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1538184094 >> 172.21.15.70:0/1538184094 conn(0x7f3fcc0d2460 msgr2=0x7f3fcc0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1538184094 shutdown_connections 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.525+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1538184094 wait complete. 2024-04-15T15:15:33.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.526+0000 7f3fd2c69640 1 Processor -- start 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 -- start start 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fb40039e0 0x7f3fcc0cbdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1f6220 0x7f3fcc1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0ce220 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc0cdf40 con 0x7f3fcc1f6220 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fd2c69640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3fcc1655f0 con 0x7f3fb40039e0 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.527+0000 7f3fc3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50468/0 (socket says 172.21.15.70:50468) 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 -- 172.21.15.70:0/1305810600 learned_addr learned my addr 172.21.15.70:0/1305810600 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fb40039e0 0x7f3fcc0cbdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 -- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fb40039e0 msgr2=0x7f3fcc0cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.536 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fb40039e0 0x7f3fcc0cbdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fd1206640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1f6220 0x7f3fcc1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 -- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1f6220 msgr2=0x7f3fcc1f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1f6220 0x7f3fcc1f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 -- 172.21.15.70:0/1305810600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc0b7d30 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc3fff640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00e7b0 tx=0x7f3fbc00ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc00c660 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fbc018070 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc0b8010 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc00ca40 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.528+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc0af780 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.529+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fcc0689e0 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.537 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.530+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3fbc00cbe0 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.530+0000 7f3fc17fa640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 0x7f3fb806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.530+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3fbc08b910 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.530+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 0x7f3fb806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.531+0000 7f3fd0a05640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 0x7f3fb806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8033000 tx=0x7f3fc8009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.542 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.534+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3fbc059a90 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.710+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3fcc0b4e20 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.711+0000 7f3fc17fa640 1 -- 172.21.15.70:0/1305810600 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f3fbc059390 con 0x7f3fcc0b77f0 2024-04-15T15:15:33.713 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 msgr2=0x7f3fb806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 0x7f3fb806ede0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8033000 tx=0x7f3fc8009030 comp rx=0 tx=0).stop 2024-04-15T15:15:33.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 msgr2=0x7f3fcc0b5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc00e7b0 tx=0x7f3fbc00ebb0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 shutdown_connections 2024-04-15T15:15:33.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3fcc1f6220 0x7f3fcc1f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3fb806c920 0x7f3fb806ede0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3fcc0b77f0 0x7f3fcc0b5190 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 --2- 172.21.15.70:0/1305810600 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3fb40039e0 0x7f3fcc0cbdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 >> 172.21.15.70:0/1305810600 conn(0x7f3fcc0d2460 msgr2=0x7f3fcc09b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 shutdown_connections 2024-04-15T15:15:33.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.713+0000 7f3fd2c69640 1 -- 172.21.15.70:0/1305810600 wait complete. 2024-04-15T15:15:33.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:33.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.830+0000 7fa952f0e640 1 Processor -- start 2024-04-15T15:15:33.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.830+0000 7fa952f0e640 1 -- start start 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa952f0e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa94c070940 0x7fa94c09b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa952f0e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa94c0d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa952f0e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa94c0d05e0 con 0x7fa94c160310 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa952f0e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa94c15f720 con 0x7fa94c070940 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa952f0e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c15fa00 con 0x7fa94c0d0da0 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa943fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa94c160310 0x7fa94c070d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43266/0 (socket says 172.21.15.70:43266) 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa943fff640 1 -- 172.21.15.70:0/3387992765 learned_addr learned my addr 172.21.15.70:0/3387992765 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:33.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9514ab640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa94c0d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa950caa640 1 --2- 172.21.15.70:0/3387992765 >> v2:172.21.15.96:3300/0 conn(0x7fa94c070940 0x7fa94c09b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2781805104 0 0) 0x7fa94c0d05e0 con 0x7fa94c160310 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa934003520 con 0x7fa94c160310 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3791277821 0 0) 0x7fa934003520 con 0x7fa94c160310 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 msgr2=0x7fa94c0d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa94c0d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 >> v2:172.21.15.96:3300/0 conn(0x7fa94c070940 msgr2=0x7fa94c09b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 --2- 172.21.15.70:0/3387992765 >> v2:172.21.15.96:3300/0 conn(0x7fa94c070940 0x7fa94c09b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c15fce0 con 0x7fa94c160310 2024-04-15T15:15:33.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fa93c001670 con 0x7fa94c160310 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 >> v1:172.21.15.70:6789/0 conn(0x7fa94c160310 legacy=0x7fa94c070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa9340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa934004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa934004d40 0x7fa9340051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c15f720 con 0x7fa934003f30 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c15fa00 con 0x7fa934004d40 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.831+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa93400b9c0 con 0x7fa94c0d0da0 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa934004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa950caa640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa9340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa9514ab640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa934004d40 0x7fa9340051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 -- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 msgr2=0x7fa9340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c0d0da0 0x7fa9340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 -- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa934004d40 msgr2=0x7fa9340051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa934004d40 0x7fa9340051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa943fff640 1 -- 172.21.15.70:0/3387992765 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa934003520 con 0x7fa934003f30 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.832+0000 7fa9514ab640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa934004d40 0x7fa9340051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa943fff640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa934004800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c210 tx=0x7fa934005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75165bcb49716bac server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c007d50 con 0x7fa934003f30 2024-04-15T15:15:33.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa93c030070 con 0x7fa934003f30 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa9437fe640 1 -- 172.21.15.70:0/3387992765 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa93c02cdb0 con 0x7fa934003f30 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 -- 172.21.15.70:0/3387992765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 msgr2=0x7fa934004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3387992765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa934004800 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa93c00c210 tx=0x7fa934005830 comp rx=0 tx=0).stop 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa9514ab640 1 -- 172.21.15.70:0/3387992765 reap_dead start 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 -- 172.21.15.70:0/3387992765 shutdown_connections 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 -- 172.21.15.70:0/3387992765 >> 172.21.15.70:0/3387992765 conn(0x7fa94c0b3f40 msgr2=0x7fa94c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 -- 172.21.15.70:0/3387992765 shutdown_connections 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.833+0000 7fa952f0e640 1 -- 172.21.15.70:0/3387992765 wait complete. 2024-04-15T15:15:33.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.834+0000 7fa952f0e640 1 Processor -- start 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 -- start start 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c09f2b0 0x7fa94c0b52a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c0b57e0 0x7fa94c0c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c0d1230 con 0x7fa934003f30 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa94c160770 con 0x7fa94c0b57e0 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa952f0e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa94c0d13b0 con 0x7fa94c09f2b0 2024-04-15T15:15:33.844 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa950caa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa950caa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50478/0 (socket says 172.21.15.70:50478) 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.835+0000 7fa950caa640 1 -- 172.21.15.70:0/3752564293 learned_addr learned my addr 172.21.15.70:0/3752564293 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 -- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c09f2b0 msgr2=0x7fa94c0b52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa943fff640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c09f2b0 0x7fa94c0b52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa9514ab640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c0b57e0 0x7fa94c0c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c09f2b0 0x7fa94c0b52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 -- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c0b57e0 msgr2=0x7fa94c0c7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c0b57e0 0x7fa94c0c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 -- 172.21.15.70:0/3752564293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa94c15a130 con 0x7fa934003f30 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa950caa640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa948011e30 tx=0x7fa94800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948015630 con 0x7fa934003f30 2024-04-15T15:15:33.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9480157d0 con 0x7fa934003f30 2024-04-15T15:15:33.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948015ba0 con 0x7fa934003f30 2024-04-15T15:15:33.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa94c0b5c60 con 0x7fa934003f30 2024-04-15T15:15:33.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.836+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa94c0b6220 con 0x7fa934003f30 2024-04-15T15:15:33.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.837+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa94c0689e0 con 0x7fa934003f30 2024-04-15T15:15:33.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.838+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa94801d400 con 0x7fa934003f30 2024-04-15T15:15:33.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.838+0000 7fa9417fa640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 0x7fa93806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:33.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.838+0000 7fa943fff640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 0x7fa93806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:33.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.838+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa94808d120 con 0x7fa934003f30 2024-04-15T15:15:33.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.838+0000 7fa943fff640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 0x7fa93806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa93c034000 tx=0x7fa93c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:33.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:33.842+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa94c15a130 con 0x7fa934003f30 2024-04-15T15:15:34.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.016+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa94c070720 con 0x7fa934003f30 2024-04-15T15:15:34.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.017+0000 7fa9417fa640 1 -- 172.21.15.70:0/3752564293 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fa948019030 con 0x7fa934003f30 2024-04-15T15:15:34.019 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:34.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.018+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 msgr2=0x7fa93806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.018+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 0x7fa93806edb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa93c034000 tx=0x7fa93c007000 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 msgr2=0x7fa94c159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa948011e30 tx=0x7fa94800f580 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 shutdown_connections 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa94c0b57e0 0x7fa94c0c7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa94c09f2b0 0x7fa94c0b52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa93806c8f0 0x7fa93806edb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 --2- 172.21.15.70:0/3752564293 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa934003f30 0x7fa94c159bf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 >> 172.21.15.70:0/3752564293 conn(0x7fa94c0b3f40 msgr2=0x7fa94c0b4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 shutdown_connections 2024-04-15T15:15:34.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.019+0000 7fa952f0e640 1 -- 172.21.15.70:0/3752564293 wait complete. 2024-04-15T15:15:34.032 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:34.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.133+0000 7febe1796640 1 Processor -- start 2024-04-15T15:15:34.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.133+0000 7febe1796640 1 -- start start 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febe1796640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febdc0d2360 0x7febdc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febe1796640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febe1796640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febdc0b47a0 con 0x7febdc15e700 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febe1796640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7febdc0b53b0 con 0x7febdc0d2360 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febe1796640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0b4a80 con 0x7febdc15eae0 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febdc0d2360 0x7febdc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febdc0d2360 0x7febdc0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50480/0 (socket says 172.21.15.70:50480) 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 -- 172.21.15.70:0/430970280 learned_addr learned my addr 172.21.15.70:0/430970280 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febdb7fe640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 -- 172.21.15.70:0/430970280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 msgr2=0x7febdc0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 -- 172.21.15.70:0/430970280 >> v1:172.21.15.96:6789/0 conn(0x7febdc15e700 legacy=0x7febdc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.134+0000 7febda7fc640 1 -- 172.21.15.70:0/430970280 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc0b4d60 con 0x7febdc0d2360 2024-04-15T15:15:34.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.135+0000 7febda7fc640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc0d2b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7febdc08c7b0 tx=0x7febd002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6b979867ad0611f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.135+0000 7febd9ffb640 1 -- 172.21.15.70:0/430970280 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0035020 con 0x7febdc0d2360 2024-04-15T15:15:34.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.135+0000 7febd9ffb640 1 -- 172.21.15.70:0/430970280 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febd0031830 con 0x7febdc0d2360 2024-04-15T15:15:34.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.135+0000 7febd9ffb640 1 -- 172.21.15.70:0/430970280 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febd0031c00 con 0x7febdc0d2360 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 -- 172.21.15.70:0/430970280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 msgr2=0x7febdc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc0d2b60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7febdc08c7b0 tx=0x7febd002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 -- 172.21.15.70:0/430970280 shutdown_connections 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 --2- 172.21.15.70:0/430970280 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc0d2b60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 -- 172.21.15.70:0/430970280 >> 172.21.15.70:0/430970280 conn(0x7febdc08bf60 msgr2=0x7febdc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 -- 172.21.15.70:0/430970280 shutdown_connections 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.136+0000 7febe1796640 1 -- 172.21.15.70:0/430970280 wait complete. 2024-04-15T15:15:34.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.137+0000 7febe1796640 1 Processor -- start 2024-04-15T15:15:34.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.137+0000 7febe1796640 1 -- start start 2024-04-15T15:15:34.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febdc15e700 0x7febdc15fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc1601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febdc0d1d40 con 0x7febdc0d2360 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7febdc0b4450 con 0x7febdc15e700 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febe1796640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7febdc0d1ec0 con 0x7febdc15eae0 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50486/0 (socket says 172.21.15.70:50486) 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 -- 172.21.15.70:0/3264772937 learned_addr learned my addr 172.21.15.70:0/3264772937 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febda7fc640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febdc15e700 0x7febdc15fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdb7fe640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc1601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 -- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 msgr2=0x7febdc1601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc1601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 -- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febdc15e700 msgr2=0x7febdc15fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febdc15e700 0x7febdc15fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 -- 172.21.15.70:0/3264772937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febdc161c00 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febdaffd640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7febcc00c7a0 tx=0x7febcc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.138+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc016020 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.139+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febcc00aa50 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.139+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febcc00ae40 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.139+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febdc1607d0 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.139+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febdc160d90 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.140+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febdc0cc340 con 0x7febdc0d2360 2024-04-15T15:15:34.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.140+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febcc019020 con 0x7febdc0d2360 2024-04-15T15:15:34.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.141+0000 7febc37fe640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 0x7febc806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.141+0000 7febda7fc640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 0x7febc806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.141+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7febcc08bfb0 con 0x7febdc0d2360 2024-04-15T15:15:34.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.141+0000 7febda7fc640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 0x7febc806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7febd0036000 tx=0x7febd003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.147 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.145+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febcc059ae0 con 0x7febdc0d2360 2024-04-15T15:15:34.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.361+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febdc0689e0 con 0x7febdc0d2360 2024-04-15T15:15:34.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.362+0000 7febc37fe640 1 -- 172.21.15.70:0/3264772937 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7febcc05dde0 con 0x7febdc0d2360 2024-04-15T15:15:34.364 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:34.364 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:34.364 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 msgr2=0x7febc806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 0x7febc806edb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7febd0036000 tx=0x7febd003c040 comp rx=0 tx=0).stop 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 msgr2=0x7febdc157920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7febcc00c7a0 tx=0x7febcc00cc60 comp rx=0 tx=0).stop 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 shutdown_connections 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febdc15eae0 0x7febdc1601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febdc15e700 0x7febdc15fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febc806c8f0 0x7febc806edb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 --2- 172.21.15.70:0/3264772937 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febdc0d2360 0x7febdc157920 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 >> 172.21.15.70:0/3264772937 conn(0x7febdc08bf60 msgr2=0x7febdc0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 shutdown_connections 2024-04-15T15:15:34.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.364+0000 7febe1796640 1 -- 172.21.15.70:0/3264772937 wait complete. 2024-04-15T15:15:34.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:34.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.482+0000 7fde93660640 1 Processor -- start 2024-04-15T15:15:34.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.482+0000 7fde93660640 1 -- start start 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.482+0000 7fde93660640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde93660640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde93660640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde8c15eb70 con 0x7fde8c0b7be0 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde93660640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fde8c0d3790 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde93660640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c0d45b0 con 0x7fde8c15e790 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde913fc640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde913fc640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:41124/0 (socket says 172.21.15.70:41124) 2024-04-15T15:15:34.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde913fc640 1 -- 172.21.15.70:0/2431526587 learned_addr learned my addr 172.21.15.70:0/2431526587 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:48416/0 (socket says 172.21.15.70:48416) 2024-04-15T15:15:34.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 -- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 msgr2=0x7fde8c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 -- 172.21.15.70:0/2431526587 >> v1:172.21.15.96:6789/0 conn(0x7fde8c0b7be0 legacy=0x7fde8c0d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde7ffff640 1 -- 172.21.15.70:0/2431526587 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476119294 0 0) 0x7fde8c15eb70 con 0x7fde8c0b7be0 2024-04-15T15:15:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde91bfd640 1 -- 172.21.15.70:0/2431526587 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c0d3c80 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.483+0000 7fde913fc640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.484+0000 7fde91bfd640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0d3910 tx=0x7fde8402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ff4975a66297f47 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.484+0000 7fde7ffff640 1 -- 172.21.15.70:0/2431526587 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84039070 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.484+0000 7fde7ffff640 1 -- 172.21.15.70:0/2431526587 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8402cb20 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.484+0000 7fde7ffff640 1 -- 172.21.15.70:0/2431526587 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84034440 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 -- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 msgr2=0x7fde8c0d4a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fde8c0d3910 tx=0x7fde8402bd70 comp rx=0 tx=0).stop 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 -- 172.21.15.70:0/2431526587 shutdown_connections 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c0d4a40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 --2- 172.21.15.70:0/2431526587 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c15e790 0x7fde8c0b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 -- 172.21.15.70:0/2431526587 >> 172.21.15.70:0/2431526587 conn(0x7fde8c067b40 msgr2=0x7fde8c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 -- 172.21.15.70:0/2431526587 shutdown_connections 2024-04-15T15:15:34.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.485+0000 7fde93660640 1 -- 172.21.15.70:0/2431526587 wait complete. 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.486+0000 7fde93660640 1 Processor -- start 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.486+0000 7fde93660640 1 -- start start 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c0b7be0 0x7fde8c15c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c1f3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0d4f80 con 0x7fde8c15e790 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde8c0d4240 con 0x7fde8c0d1cf0 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde93660640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fde8c0d5100 con 0x7fde8c0b7be0 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50496/0 (socket says 172.21.15.70:50496) 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 -- 172.21.15.70:0/4142729017 learned_addr learned my addr 172.21.15.70:0/4142729017 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde913fc640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c0b7be0 0x7fde8c15c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde90bfb640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c1f3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 -- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c0b7be0 msgr2=0x7fde8c15c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c0b7be0 0x7fde8c15c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 -- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 msgr2=0x7fde8c1f3dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c1f3dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.487+0000 7fde91bfd640 1 -- 172.21.15.70:0/4142729017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde8c1f4300 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde91bfd640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fde84037040 tx=0x7fde84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84044020 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8403a040 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84039050 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde8c1f45b0 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.488+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde8c1f4b70 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.489+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde8c0689e0 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.489+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fde84003bb0 con 0x7fde8c15e790 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.490+0000 7fde7dffb640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 0x7fde7006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.490+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fde840ae270 con 0x7fde8c15e790 2024-04-15T15:15:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.490+0000 7fde913fc640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 0x7fde7006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.490+0000 7fde913fc640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 0x7fde7006edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fde80006ca0 tx=0x7fde80009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.494+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fde8404e060 con 0x7fde8c15e790 2024-04-15T15:15:34.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.672+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fde8c0d1720 con 0x7fde8c15e790 2024-04-15T15:15:34.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.673+0000 7fde7dffb640 1 -- 172.21.15.70:0/4142729017 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7fde8407be60 con 0x7fde8c15e790 2024-04-15T15:15:34.675 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:34.675 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 msgr2=0x7fde7006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 0x7fde7006edb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fde80006ca0 tx=0x7fde80009240 comp rx=0 tx=0).stop 2024-04-15T15:15:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 msgr2=0x7fde8c158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fde84037040 tx=0x7fde84004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 shutdown_connections 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde7006c8f0 0x7fde7006edb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde8c15e790 0x7fde8c158a60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde8c0d1cf0 0x7fde8c1f3dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 --2- 172.21.15.70:0/4142729017 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde8c0b7be0 0x7fde8c15c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 >> 172.21.15.70:0/4142729017 conn(0x7fde8c067b40 msgr2=0x7fde8c0cd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 shutdown_connections 2024-04-15T15:15:34.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.675+0000 7fde93660640 1 -- 172.21.15.70:0/4142729017 wait complete. 2024-04-15T15:15:34.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:34.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:34.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.792+0000 7fbe6fdf4640 1 Processor -- start 2024-04-15T15:15:34.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.793+0000 7fbe6fdf4640 1 -- start start 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.793+0000 7fbe6fdf4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe680d0690 0x7fbe680d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.793+0000 7fbe6fdf4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbe680b3160 0x7fbe6815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6fdf4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe680d0fb0 con 0x7fbe6815ef70 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6fdf4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbe6815cb50 con 0x7fbe680d0690 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6fdf4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbe6815bfd0 con 0x7fbe680b3160 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe680d0690 0x7fbe680d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbe680d0690 0x7fbe680d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50512/0 (socket says 172.21.15.70:50512) 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 -- 172.21.15.70:0/1447551471 learned_addr learned my addr 172.21.15.70:0/1447551471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6d38f640 1 --2- 172.21.15.70:0/1447551471 >> v2:172.21.15.96:3300/0 conn(0x7fbe680b3160 0x7fbe6815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 -- 172.21.15.70:0/1447551471 >> v2:172.21.15.96:3300/0 conn(0x7fbe680b3160 msgr2=0x7fbe6815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 --2- 172.21.15.70:0/1447551471 >> v2:172.21.15.96:3300/0 conn(0x7fbe680b3160 0x7fbe6815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 -- 172.21.15.70:0/1447551471 >> v1:172.21.15.96:6789/0 conn(0x7fbe6815ef70 legacy=0x7fbe6815f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:34.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.794+0000 7fbe6db90640 1 -- 172.21.15.70:0/1447551471 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6815c2b0 con 0x7fbe680d0690 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6db90640 1 --2- 172.21.15.70:0/1447551471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe680d0690 0x7fbe680d0a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe6806fec0 tx=0x7fbe5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634e8c2a666fb03f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6cb8e640 1 -- 172.21.15.70:0/1447551471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c035020 con 0x7fbe680d0690 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6cb8e640 1 -- 172.21.15.70:0/1447551471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe5c031870 con 0x7fbe680d0690 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6cb8e640 1 -- 172.21.15.70:0/1447551471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c031c40 con 0x7fbe680d0690 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/1447551471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe680d0690 msgr2=0x7fbe680d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/1447551471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe680d0690 0x7fbe680d0a70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe6806fec0 tx=0x7fbe5c02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/1447551471 shutdown_connections 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/1447551471 >> v2:172.21.15.96:3300/0 conn(0x7fbe680b3160 0x7fbe6815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/1447551471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe680d0690 0x7fbe680d0a70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.795+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/1447551471 >> 172.21.15.70:0/1447551471 conn(0x7fbe6806be00 msgr2=0x7fbe6806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.796+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/1447551471 shutdown_connections 2024-04-15T15:15:34.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.796+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/1447551471 wait complete. 2024-04-15T15:15:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 Processor -- start 2024-04-15T15:15:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 -- start start 2024-04-15T15:15:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 0x7fbe681524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 0x7fbe68156320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe680b4fb0 con 0x7fbe6815ef70 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6815c870 con 0x7fbe680b3160 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6fdf4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbe6815bbd0 con 0x7fbe680d0690 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6e391640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6e391640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50528/0 (socket says 172.21.15.70:50528) 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.797+0000 7fbe6e391640 1 -- 172.21.15.70:0/2373436225 learned_addr learned my addr 172.21.15.70:0/2373436225 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6db90640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 0x7fbe681524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6d38f640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 0x7fbe68156320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 -- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 msgr2=0x7fbe68156320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 0x7fbe68156320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 -- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 msgr2=0x7fbe681524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 0x7fbe681524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 -- 172.21.15.70:0/2373436225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe68154ad0 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6db90640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 0x7fbe681524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6d38f640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 0x7fbe68156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6e391640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c780 tx=0x7fbe6000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60017070 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe60003da0 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe68154db0 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe600052b0 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.798+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe68155e00 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.799+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe68156000 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.800+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbe6001a070 con 0x7fbe6815ef70 2024-04-15T15:15:34.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.800+0000 7fbe5a7fc640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 0x7fbe4c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:34.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.800+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbe6008b4b0 con 0x7fbe6815ef70 2024-04-15T15:15:34.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.800+0000 7fbe6db90640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 0x7fbe4c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:34.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.800+0000 7fbe6db90640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 0x7fbe4c06edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c036000 tx=0x7fbe5c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:34.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.804+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbe68156000 con 0x7fbe6815ef70 2024-04-15T15:15:34.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.979+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbe680689e0 con 0x7fbe6815ef70 2024-04-15T15:15:34.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.980+0000 7fbe5a7fc640 1 -- 172.21.15.70:0/2373436225 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7fbe60059630 con 0x7fbe6815ef70 2024-04-15T15:15:34.982 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:34.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 msgr2=0x7fbe4c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 0x7fbe4c06edb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c036000 tx=0x7fbe5c008030 comp rx=0 tx=0).stop 2024-04-15T15:15:34.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 msgr2=0x7fbe681540f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000c780 tx=0x7fbe6000cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 shutdown_connections 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbe4c06c8f0 0x7fbe4c06edb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe6815ef70 0x7fbe681540f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbe680d0690 0x7fbe68156320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 --2- 172.21.15.70:0/2373436225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbe680b3160 0x7fbe681524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 >> 172.21.15.70:0/2373436225 conn(0x7fbe6806be00 msgr2=0x7fbe68153860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 shutdown_connections 2024-04-15T15:15:34.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:34.982+0000 7fbe6fdf4640 1 -- 172.21.15.70:0/2373436225 wait complete. 2024-04-15T15:15:34.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:35.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.100+0000 7f34dec77640 1 Processor -- start 2024-04-15T15:15:35.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.100+0000 7f34dec77640 1 -- start start 2024-04-15T15:15:35.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dec77640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f34d80d3240 0x7f34d80d0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dec77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 0x7f34d80b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dec77640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34d80d2480 con 0x7f34d815e790 2024-04-15T15:15:35.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dec77640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f34d80d2db0 con 0x7f34d80d3240 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dec77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f34d80d2760 con 0x7f34d80d1440 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34cffff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f34d80d3240 0x7f34d80d0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dd214640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 0x7f34d80b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dca13640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f34d815e790 0x7f34d80cfd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:43956/0 (socket says 172.21.15.70:43956) 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.101+0000 7f34dca13640 1 -- 172.21.15.70:0/1306138663 learned_addr learned my addr 172.21.15.70:0/1306138663 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:35.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34cf7fe640 1 -- 172.21.15.70:0/1306138663 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 821264258 0 0) 0x7f34d80d2480 con 0x7f34d815e790 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34cf7fe640 1 -- 172.21.15.70:0/1306138663 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f34c0003520 con 0x7f34d815e790 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34dd214640 1 -- 172.21.15.70:0/1306138663 >> v2:172.21.15.96:3300/0 conn(0x7f34d80d3240 msgr2=0x7f34d80d0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34dd214640 1 --2- 172.21.15.70:0/1306138663 >> v2:172.21.15.96:3300/0 conn(0x7f34d80d3240 0x7f34d80d0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34dd214640 1 -- 172.21.15.70:0/1306138663 >> v1:172.21.15.96:6789/0 conn(0x7f34d815e790 legacy=0x7f34d80cfd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34dd214640 1 -- 172.21.15.70:0/1306138663 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d80d2a40 con 0x7f34d80d1440 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.102+0000 7f34dd214640 1 --2- 172.21.15.70:0/1306138663 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 0x7f34d80b74f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009d90 tx=0x7f34d002bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25dc4e24adabee2f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34cf7fe640 1 -- 172.21.15.70:0/1306138663 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0039070 con 0x7f34d80d1440 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34cf7fe640 1 -- 172.21.15.70:0/1306138663 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d002cc50 con 0x7f34d80d1440 2024-04-15T15:15:35.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34cf7fe640 1 -- 172.21.15.70:0/1306138663 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d0034440 con 0x7f34d80d1440 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 -- 172.21.15.70:0/1306138663 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 msgr2=0x7f34d80b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 --2- 172.21.15.70:0/1306138663 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 0x7f34d80b74f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f34d0009d90 tx=0x7f34d002bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 -- 172.21.15.70:0/1306138663 shutdown_connections 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 --2- 172.21.15.70:0/1306138663 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d1440 0x7f34d80b74f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 --2- 172.21.15.70:0/1306138663 >> v2:172.21.15.96:3300/0 conn(0x7f34d80d3240 0x7f34d80d0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 -- 172.21.15.70:0/1306138663 >> 172.21.15.70:0/1306138663 conn(0x7f34d808bf60 msgr2=0x7f34d80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 -- 172.21.15.70:0/1306138663 shutdown_connections 2024-04-15T15:15:35.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.103+0000 7f34dec77640 1 -- 172.21.15.70:0/1306138663 wait complete. 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.104+0000 7f34dec77640 1 Processor -- start 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.104+0000 7f34dec77640 1 -- start start 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d3240 0x7f34d814c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d815e790 0x7f34d80cd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f34d80d5180 con 0x7f34d80d1440 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f34d80d0430 con 0x7f34d815e790 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dec77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f34d80d5300 con 0x7f34d80d3240 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dca13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dca13640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50538/0 (socket says 172.21.15.70:50538) 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.105+0000 7f34dca13640 1 -- 172.21.15.70:0/44429204 learned_addr learned my addr 172.21.15.70:0/44429204 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34cffff640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d3240 0x7f34d814c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dd214640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d815e790 0x7f34d80cd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 -- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d3240 msgr2=0x7f34d814c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d3240 0x7f34d814c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 -- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d815e790 msgr2=0x7f34d80cd0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d815e790 0x7f34d80cd0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 -- 172.21.15.70:0/44429204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34d814c810 con 0x7f34d80d1440 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dca13640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c390 tx=0x7f34d400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d4017070 con 0x7f34d80d1440 2024-04-15T15:15:35.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d4005ce0 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34d814cab0 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d40072f0 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.106+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34d814d070 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.107+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34d80689e0 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.108+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f34d401a070 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.108+0000 7f34cd7fa640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 0x7f34c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.108+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f34d408b280 con 0x7f34d80d1440 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.108+0000 7f34cffff640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 0x7f34c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.109+0000 7f34cffff640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 0x7f34c406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34d80ccdc0 tx=0x7f34c8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.112+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f34d4058ca0 con 0x7f34d80d1440 2024-04-15T15:15:35.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.289+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f34d80d1ce0 con 0x7f34d80d1440 2024-04-15T15:15:35.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.289+0000 7f34cd7fa640 1 -- 172.21.15.70:0/44429204 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f34d405cfa0 con 0x7f34d80d1440 2024-04-15T15:15:35.291 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 msgr2=0x7f34c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 0x7f34c406edb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f34d80ccdc0 tx=0x7f34c8009190 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 msgr2=0x7f34d80cb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f34d400c390 tx=0x7f34d400c850 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 shutdown_connections 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34d815e790 0x7f34d80cd0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f34d80d3240 0x7f34d814c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f34c406c8f0 0x7f34c406edb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 --2- 172.21.15.70:0/44429204 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34d80d1440 0x7f34d80cb200 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 >> 172.21.15.70:0/44429204 conn(0x7f34d808bf60 msgr2=0x7f34d806d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 shutdown_connections 2024-04-15T15:15:35.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.292+0000 7f34dec77640 1 -- 172.21.15.70:0/44429204 wait complete. 2024-04-15T15:15:35.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:35.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.411+0000 7f9b5c80d640 1 Processor -- start 2024-04-15T15:15:35.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.411+0000 7f9b5c80d640 1 -- start start 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5c80d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9b580701f0 0x7f9b58160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5c80d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b580d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5c80d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b58070720 con 0x7f9b5809a9d0 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5c80d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9b5809ba90 con 0x7f9b580701f0 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5c80d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9b580d08c0 con 0x7f9b5815fb30 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5659c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9b5809a9d0 0x7f9b5809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43280/0 (socket says 172.21.15.70:43280) 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b5659c640 1 -- 172.21.15.70:0/397093020 learned_addr learned my addr 172.21.15.70:0/397093020 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b56d9d640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b580d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.412+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/397093020 >> v2:172.21.15.96:3300/0 conn(0x7f9b580701f0 0x7f9b58160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2983779361 0 0) 0x7f9b58070720 con 0x7f9b5809a9d0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9b40003520 con 0x7f9b5809a9d0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 964274229 0 0) 0x7f9b40003520 con 0x7f9b5809a9d0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 msgr2=0x7f9b580d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b580d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 >> v2:172.21.15.96:3300/0 conn(0x7f9b580701f0 msgr2=0x7f9b58160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 --2- 172.21.15.70:0/397093020 >> v2:172.21.15.96:3300/0 conn(0x7f9b580701f0 0x7f9b58160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b58070a90 con 0x7f9b5809a9d0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f9b480019f0 con 0x7f9b5809a9d0 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 >> v1:172.21.15.70:6789/0 conn(0x7f9b5809a9d0 legacy=0x7f9b5809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b400039b0 0x7f9b40003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b40004800 0x7f9b40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b580d08c0 con 0x7f9b40004800 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9b4000b9c0 con 0x7f9b400039b0 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9b4000bb40 con 0x7f9b5815fb30 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b40004800 0x7f9b40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5659c640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b400039b0 0x7f9b40003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b56d9d640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 -- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 msgr2=0x7f9b400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b5815fb30 0x7f9b400051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 -- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b400039b0 msgr2=0x7f9b40003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b400039b0 0x7f9b40003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b55d9b640 1 -- 172.21.15.70:0/397093020 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b40003520 con 0x7f9b40004800 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.413+0000 7f9b5659c640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b400039b0 0x7f9b40003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.414+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b40004800 0x7f9b40004c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009a90 tx=0x7f9b40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=832084a00b46884c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.414+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c039070 con 0x7f9b40004800 2024-04-15T15:15:35.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.414+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4c02c6b0 con 0x7f9b40004800 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.414+0000 7f9b5559a640 1 -- 172.21.15.70:0/397093020 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c009690 con 0x7f9b40004800 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 -- 172.21.15.70:0/397093020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b40004800 msgr2=0x7f9b40004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/397093020 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b40004800 0x7f9b40004c80 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c009a90 tx=0x7f9b40005830 comp rx=0 tx=0).stop 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b56d9d640 1 -- 172.21.15.70:0/397093020 reap_dead start 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 -- 172.21.15.70:0/397093020 shutdown_connections 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 -- 172.21.15.70:0/397093020 >> 172.21.15.70:0/397093020 conn(0x7f9b580d2460 msgr2=0x7f9b580d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 -- 172.21.15.70:0/397093020 shutdown_connections 2024-04-15T15:15:35.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.415+0000 7f9b5c80d640 1 -- 172.21.15.70:0/397093020 wait complete. 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.416+0000 7f9b5c80d640 1 Processor -- start 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.416+0000 7f9b5c80d640 1 -- start start 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b40004800 0x7f9b58157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b580b72b0 0x7f9b580b7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5815ffc0 con 0x7f9b580b5b40 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9b58071050 con 0x7f9b40004800 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5c80d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9b580b1c60 con 0x7f9b580b72b0 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50562/0 (socket says 172.21.15.70:50562) 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 -- 172.21.15.70:0/3211840085 learned_addr learned my addr 172.21.15.70:0/3211840085 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b5659c640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b40004800 0x7f9b58157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b56d9d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b580b72b0 0x7f9b580b7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 -- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b580b72b0 msgr2=0x7f9b580b7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b580b72b0 0x7f9b580b7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 -- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b40004800 msgr2=0x7f9b58157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b40004800 0x7f9b58157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 -- 172.21.15.70:0/3211840085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5815c470 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.417+0000 7f9b55d9b640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0099e0 tx=0x7f9b4c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.418+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c039050 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.418+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b4c03a040 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.418+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5815c750 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.418+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4c0044e0 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.418+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5815cd10 con 0x7f9b580b5b40 2024-04-15T15:15:35.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.419+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b5815d510 con 0x7f9b580b5b40 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.419+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9b4c047020 con 0x7f9b580b5b40 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.419+0000 7f9b46ffd640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 0x7f9b3806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.420+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f9b4c0ad850 con 0x7f9b580b5b40 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.420+0000 7f9b5659c640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 0x7f9b3806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.420+0000 7f9b5659c640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 0x7f9b3806ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b48001330 tx=0x7f9b48009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.424+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9b4c07ba60 con 0x7f9b580b5b40 2024-04-15T15:15:35.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.597+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9b580690f0 con 0x7f9b580b5b40 2024-04-15T15:15:35.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.597+0000 7f9b46ffd640 1 -- 172.21.15.70:0/3211840085 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f9b580690f0 con 0x7f9b580b5b40 2024-04-15T15:15:35.599 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:35.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.599+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 msgr2=0x7f9b3806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.599+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 0x7f9b3806ede0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9b48001330 tx=0x7f9b48009040 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.599+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 msgr2=0x7f9b5815bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.599+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9b4c0099e0 tx=0x7f9b4c004730 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 shutdown_connections 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9b580b72b0 0x7f9b580b7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9b3806c920 0x7f9b3806ede0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9b580b5b40 0x7f9b5815bb40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 --2- 172.21.15.70:0/3211840085 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9b40004800 0x7f9b58157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 >> 172.21.15.70:0/3211840085 conn(0x7f9b580d2460 msgr2=0x7f9b580b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 shutdown_connections 2024-04-15T15:15:35.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.600+0000 7f9b5c80d640 1 -- 172.21.15.70:0/3211840085 wait complete. 2024-04-15T15:15:35.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:35.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 Processor -- start 2024-04-15T15:15:35.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 -- start start 2024-04-15T15:15:35.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd18015c8d0 0x7fd1800cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 0x7fd18009a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd18009af10 0x7fd1800d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd1800b5b00 con 0x7fd18009af10 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd1800d0350 con 0x7fd18015c8d0 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.717+0000 7fd184d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd1800ce100 con 0x7fd18009b650 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd18009af10 0x7fd1800d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd18009af10 0x7fd1800d0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50574/0 (socket says 172.21.15.70:50574) 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 -- 172.21.15.70:0/267659733 learned_addr learned my addr 172.21.15.70:0/267659733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd177fff640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 0x7fd18009a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17e59c640 1 --2- 172.21.15.70:0/267659733 >> v2:172.21.15.96:3300/0 conn(0x7fd18015c8d0 0x7fd1800cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 -- 172.21.15.70:0/267659733 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 msgr2=0x7fd18009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 0x7fd18009a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 -- 172.21.15.70:0/267659733 >> v2:172.21.15.96:3300/0 conn(0x7fd18015c8d0 msgr2=0x7fd1800cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 --2- 172.21.15.70:0/267659733 >> v2:172.21.15.96:3300/0 conn(0x7fd18015c8d0 0x7fd1800cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd17ed9d640 1 -- 172.21.15.70:0/267659733 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1800b54b0 con 0x7fd18009af10 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.718+0000 7fd177fff640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 0x7fd18009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:35.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd17ed9d640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009af10 0x7fd1800d0540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd18009b620 tx=0x7fd17802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce27ca33f4324d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd17dd9b640 1 -- 172.21.15.70:0/267659733 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd17802c9e0 con 0x7fd18009af10 2024-04-15T15:15:35.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd17dd9b640 1 -- 172.21.15.70:0/267659733 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd17802cb80 con 0x7fd18009af10 2024-04-15T15:15:35.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd17dd9b640 1 -- 172.21.15.70:0/267659733 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1780343f0 con 0x7fd18009af10 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 -- 172.21.15.70:0/267659733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009af10 msgr2=0x7fd1800d0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009af10 0x7fd1800d0540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd18009b620 tx=0x7fd17802be40 comp rx=0 tx=0).stop 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 -- 172.21.15.70:0/267659733 shutdown_connections 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009af10 0x7fd1800d0540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 --2- 172.21.15.70:0/267659733 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009b650 0x7fd18009a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 --2- 172.21.15.70:0/267659733 >> v2:172.21.15.96:3300/0 conn(0x7fd18015c8d0 0x7fd1800cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 -- 172.21.15.70:0/267659733 >> 172.21.15.70:0/267659733 conn(0x7fd1800b3fb0 msgr2=0x7fd1800b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 -- 172.21.15.70:0/267659733 shutdown_connections 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.719+0000 7fd184d1c640 1 -- 172.21.15.70:0/267659733 wait complete. 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 Processor -- start 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 -- start start 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009af10 0x7fd1800c6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd180154a60 0x7fd1800712b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1800d0c00 con 0x7fd18009b650 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd1800d0a80 con 0x7fd180154a60 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.721+0000 7fd184d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd1800d0d80 con 0x7fd18009af10 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50590/0 (socket says 172.21.15.70:50590) 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 -- 172.21.15.70:0/4277043694 learned_addr learned my addr 172.21.15.70:0/4277043694 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd17e59c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009af10 0x7fd1800c6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd17ed9d640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd180154a60 0x7fd1800712b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 -- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009af10 msgr2=0x7fd1800c6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009af10 0x7fd1800c6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 -- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd180154a60 msgr2=0x7fd1800712b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd180154a60 0x7fd1800712b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 -- 172.21.15.70:0/4277043694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd18006caf0 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd177fff640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd16800e730 tx=0x7fd16800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16800f040 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd168004230 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16800ee40 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd18006cdd0 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.722+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd18006d390 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.723+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1800690f0 con 0x7fd18009b650 2024-04-15T15:15:35.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.724+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd16800c6d0 con 0x7fd18009b650 2024-04-15T15:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.724+0000 7fd176ffd640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 0x7fd16006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.724+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd16808bb90 con 0x7fd18009b650 2024-04-15T15:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.724+0000 7fd17e59c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 0x7fd16006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:35.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.724+0000 7fd17e59c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 0x7fd16006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd170007920 tx=0x7fd170008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:35.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.728+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd168017020 con 0x7fd18009b650 2024-04-15T15:15:35.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.902+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd1800cecb0 con 0x7fd18009b650 2024-04-15T15:15:35.905 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.903+0000 7fd176ffd640 1 -- 172.21.15.70:0/4277043694 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fd168059630 con 0x7fd18009b650 2024-04-15T15:15:35.905 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 msgr2=0x7fd16006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 0x7fd16006edb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd170007920 tx=0x7fd170008040 comp rx=0 tx=0).stop 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 msgr2=0x7fd18006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd16800e730 tx=0x7fd16800ebf0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 shutdown_connections 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd180154a60 0x7fd1800712b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd16006c8f0 0x7fd16006edb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd18009b650 0x7fd18006e560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 --2- 172.21.15.70:0/4277043694 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd18009af10 0x7fd1800c6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:35.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 >> 172.21.15.70:0/4277043694 conn(0x7fd1800b3fb0 msgr2=0x7fd1800ceed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:35.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 shutdown_connections 2024-04-15T15:15:35.908 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:35.905+0000 7fd184d1c640 1 -- 172.21.15.70:0/4277043694 wait complete. 2024-04-15T15:15:35.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.018+0000 7f1abf822640 1 Processor -- start 2024-04-15T15:15:36.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.018+0000 7f1abf822640 1 -- start start 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1ab815ecb0 0x7f1ab80b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1ab80b4350 0x7f1ab80b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab80b1b60 0x7f1ab80ce410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab815f090 con 0x7f1ab815ecb0 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab80cdf80 con 0x7f1ab80b4350 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abf822640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab80b10d0 con 0x7f1ab80b1b60 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abd5be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1ab815ecb0 0x7f1ab80b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abd5be640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1ab815ecb0 0x7f1ab80b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50604/0 (socket says 172.21.15.70:50604) 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abd5be640 1 -- 172.21.15.70:0/3962092962 learned_addr learned my addr 172.21.15.70:0/3962092962 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.019+0000 7f1abddbf640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab80b1b60 0x7f1ab80ce410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abcdbd640 1 --2- 172.21.15.70:0/3962092962 >> v2:172.21.15.96:3300/0 conn(0x7f1ab80b4350 0x7f1ab80b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 -- 172.21.15.70:0/3962092962 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab80b1b60 msgr2=0x7f1ab80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab80b1b60 0x7f1ab80ce410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 -- 172.21.15.70:0/3962092962 >> v2:172.21.15.96:3300/0 conn(0x7f1ab80b4350 msgr2=0x7f1ab80b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3962092962 >> v2:172.21.15.96:3300/0 conn(0x7f1ab80b4350 0x7f1ab80b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 -- 172.21.15.70:0/3962092962 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab80af920 con 0x7f1ab815ecb0 2024-04-15T15:15:36.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab815ecb0 0x7f1ab80b0b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806a050 tx=0x7f1ab402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a60a56f94a3c7a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1aaffff640 1 -- 172.21.15.70:0/3962092962 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4034070 con 0x7f1ab815ecb0 2024-04-15T15:15:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.020+0000 7f1aaffff640 1 -- 172.21.15.70:0/3962092962 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab4030860 con 0x7f1ab815ecb0 2024-04-15T15:15:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1aaffff640 1 -- 172.21.15.70:0/3962092962 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030c30 con 0x7f1ab815ecb0 2024-04-15T15:15:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 -- 172.21.15.70:0/3962092962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab815ecb0 msgr2=0x7f1ab80b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab815ecb0 0x7f1ab80b0b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ab806a050 tx=0x7f1ab402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 -- 172.21.15.70:0/3962092962 shutdown_connections 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab80b1b60 0x7f1ab80ce410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 --2- 172.21.15.70:0/3962092962 >> v2:172.21.15.96:3300/0 conn(0x7f1ab80b4350 0x7f1ab80b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 --2- 172.21.15.70:0/3962092962 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab815ecb0 0x7f1ab80b0b90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 -- 172.21.15.70:0/3962092962 >> 172.21.15.70:0/3962092962 conn(0x7f1ab80b6870 msgr2=0x7f1ab80b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 -- 172.21.15.70:0/3962092962 shutdown_connections 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.021+0000 7f1abf822640 1 -- 172.21.15.70:0/3962092962 wait complete. 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.022+0000 7f1abf822640 1 Processor -- start 2024-04-15T15:15:36.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.022+0000 7f1abf822640 1 -- start start 2024-04-15T15:15:36.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 0x7f1ab8158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab815ecb0 0x7f1ab815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80b47c0 con 0x7f1ab80b4350 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab80afee0 con 0x7f1ab80b1b60 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abf822640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab815bfd0 con 0x7f1ab815ecb0 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50616/0 (socket says 172.21.15.70:50616) 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 -- 172.21.15.70:0/3880556638 learned_addr learned my addr 172.21.15.70:0/3880556638 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 0x7f1ab8158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abddbf640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab815ecb0 0x7f1ab815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 -- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab815ecb0 msgr2=0x7f1ab815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab815ecb0 0x7f1ab815ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 -- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 msgr2=0x7f1ab8158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 0x7f1ab8158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abcdbd640 1 -- 172.21.15.70:0/3880556638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab81592e0 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.023+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 0x7f1ab8158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1abcdbd640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800d800 tx=0x7f1aa800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800b6c0 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa8019070 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab81595c0 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800ba80 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.024+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab8159b80 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.025+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab80689e0 con 0x7f1ab80b4350 2024-04-15T15:15:36.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.025+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1aa800bc20 con 0x7f1ab80b4350 2024-04-15T15:15:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.026+0000 7f1aadffb640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 0x7f1a9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.026+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1aa808c2a0 con 0x7f1ab80b4350 2024-04-15T15:15:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.026+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 0x7f1a9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.026+0000 7f1abd5be640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 0x7f1a9c06edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4036000 tx=0x7f1ab403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.030+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1aa805a420 con 0x7f1ab80b4350 2024-04-15T15:15:36.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.242+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ab80d1b70 con 0x7f1ab80b4350 2024-04-15T15:15:36.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.242+0000 7f1aadffb640 1 -- 172.21.15.70:0/3880556638 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f1aa8059d20 con 0x7f1ab80b4350 2024-04-15T15:15:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:36.244 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:36.244 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:15:36.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 msgr2=0x7f1a9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 0x7f1a9c06edb0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4036000 tx=0x7f1ab403b040 comp rx=0 tx=0).stop 2024-04-15T15:15:36.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 msgr2=0x7f1ab8154830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800d800 tx=0x7f1aa800dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 shutdown_connections 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1ab815ecb0 0x7f1ab815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a9c06c8f0 0x7f1a9c06edb0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1ab80b4350 0x7f1ab8154830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 --2- 172.21.15.70:0/3880556638 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1ab80b1b60 0x7f1ab8158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 >> 172.21.15.70:0/3880556638 conn(0x7f1ab80b6870 msgr2=0x7f1ab80ce410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 shutdown_connections 2024-04-15T15:15:36.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.244+0000 7f1abf822640 1 -- 172.21.15.70:0/3880556638 wait complete. 2024-04-15T15:15:36.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:36.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.358+0000 7f0c210c5640 1 Processor -- start 2024-04-15T15:15:36.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.358+0000 7f0c210c5640 1 -- start start 2024-04-15T15:15:36.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c210c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 0x7f0c1c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c210c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c1c0b53c0 0x7f0c1c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c210c5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c1c067e30 con 0x7f0c1c15e830 2024-04-15T15:15:36.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c210c5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1c0b3c10 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c210c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c0b3ef0 con 0x7f0c1c15ec10 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1b59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c1c0b53c0 0x7f0c1c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1ad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0c1c15e830 0x7f0c1c067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43294/0 (socket says 172.21.15.70:43294) 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1ad9d640 1 -- 172.21.15.70:0/2294081001 learned_addr learned my addr 172.21.15.70:0/2294081001 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1a59c640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 0x7f0c1c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2294081001 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 msgr2=0x7f0c1c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1b59e640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 0x7f0c1c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2294081001 >> v1:172.21.15.70:6789/0 conn(0x7f0c1c15e830 legacy=0x7f0c1c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c19d9b640 1 -- 172.21.15.70:0/2294081001 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1997642692 0 0) 0x7f0c1c067e30 con 0x7f0c1c15e830 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2294081001 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c0b41d0 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.359+0000 7f0c1a59c640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 0x7f0c1c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.360+0000 7f0c1b59e640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c0b53c0 0x7f0c1c0b5850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009d90 tx=0x7f0c1402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e31f2eaa9bf8e17 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.360+0000 7f0c19d9b640 1 -- 172.21.15.70:0/2294081001 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14039070 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.360+0000 7f0c19d9b640 1 -- 172.21.15.70:0/2294081001 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c1402cc50 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.360+0000 7f0c19d9b640 1 -- 172.21.15.70:0/2294081001 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14034440 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 -- 172.21.15.70:0/2294081001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c0b53c0 msgr2=0x7f0c1c0b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c0b53c0 0x7f0c1c0b5850 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009d90 tx=0x7f0c1402bea0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 -- 172.21.15.70:0/2294081001 shutdown_connections 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c0b53c0 0x7f0c1c0b5850 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2294081001 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c15ec10 0x7f0c1c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 -- 172.21.15.70:0/2294081001 >> 172.21.15.70:0/2294081001 conn(0x7f0c1c073360 msgr2=0x7f0c1c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 -- 172.21.15.70:0/2294081001 shutdown_connections 2024-04-15T15:15:36.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.361+0000 7f0c210c5640 1 -- 172.21.15.70:0/2294081001 wait complete. 2024-04-15T15:15:36.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.362+0000 7f0c210c5640 1 Processor -- start 2024-04-15T15:15:36.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.362+0000 7f0c210c5640 1 -- start start 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c0b53c0 0x7f0c1c146590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c1c15e830 0x7f0c1c1e5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c15ff60 con 0x7f0c1c1ea700 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c1c0b4790 con 0x7f0c1c15e830 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c210c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1c1600e0 con 0x7f0c1c0b53c0 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c1b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c1b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50630/0 (socket says 172.21.15.70:50630) 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2890633523 learned_addr learned my addr 172.21.15.70:0/2890633523 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c1a59c640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c1c15e830 0x7f0c1c1e5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.363+0000 7f0c1ad9d640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c0b53c0 0x7f0c1c146590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c0b53c0 msgr2=0x7f0c1c146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c0b53c0 0x7f0c1c146590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c1c15e830 msgr2=0x7f0c1c1e5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c1c15e830 0x7f0c1c1e5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 -- 172.21.15.70:0/2890633523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c1c1eaee0 con 0x7f0c1c1ea700 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c1b59e640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009d90 tx=0x7f0c14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14039050 con 0x7f0c1c1ea700 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c14045070 con 0x7f0c1c1ea700 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1c1f7f80 con 0x7f0c1c1ea700 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1403e610 con 0x7f0c1c1ea700 2024-04-15T15:15:36.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.364+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c1c1f8540 con 0x7f0c1c1ea700 2024-04-15T15:15:36.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.366+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c1c0689e0 con 0x7f0c1c1ea700 2024-04-15T15:15:36.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.366+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c1403e7b0 con 0x7f0c1c1ea700 2024-04-15T15:15:36.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.366+0000 7f0c037fe640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 0x7f0c0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.366+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0c140ae810 con 0x7f0c1c1ea700 2024-04-15T15:15:36.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.367+0000 7f0c1ad9d640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 0x7f0c0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.367+0000 7f0c1ad9d640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 0x7f0c0806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000f60 tx=0x7f0c0c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.372+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c1407c340 con 0x7f0c1c1ea700 2024-04-15T15:15:36.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.547+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c1c06ff10 con 0x7f0c1c1ea700 2024-04-15T15:15:36.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.548+0000 7f0c037fe640 1 -- 172.21.15.70:0/2890633523 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f0c140ace20 con 0x7f0c1c1ea700 2024-04-15T15:15:36.551 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:36.551 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.550+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 msgr2=0x7f0c0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.550+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 0x7f0c0806edb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c000f60 tx=0x7f0c0c007d50 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.550+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 msgr2=0x7f0c1c0b1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.550+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009d90 tx=0x7f0c14004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 shutdown_connections 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c0806c8f0 0x7f0c0806edb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1c1ea700 0x7f0c1c0b1ad0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c1c15e830 0x7f0c1c1e5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 --2- 172.21.15.70:0/2890633523 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1c0b53c0 0x7f0c1c146590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 >> 172.21.15.70:0/2890633523 conn(0x7f0c1c073360 msgr2=0x7f0c1c06e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 shutdown_connections 2024-04-15T15:15:36.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.551+0000 7f0c210c5640 1 -- 172.21.15.70:0/2890633523 wait complete. 2024-04-15T15:15:36.564 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:15:36.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.666+0000 7f3843b15640 1 Processor -- start 2024-04-15T15:15:36.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.666+0000 7f3843b15640 1 -- start start 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f3843b15640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0701f0 0x7f383c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f3843b15640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c15fb30 0x7f383c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f3843b15640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f383c070720 con 0x7f383c09a9d0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f3843b15640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f383c09ba90 con 0x7f383c15fb30 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f3843b15640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0d08c0 con 0x7f383c0701f0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c15fb30 0x7f383c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c15fb30 0x7f383c0d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50632/0 (socket says 172.21.15.70:50632) 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 -- 172.21.15.70:0/4248902344 learned_addr learned my addr 172.21.15.70:0/4248902344 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38410b0640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0701f0 0x7f383c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 -- 172.21.15.70:0/4248902344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0701f0 msgr2=0x7f383c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0701f0 0x7f383c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.667+0000 7f38420b2640 1 -- 172.21.15.70:0/4248902344 >> v1:172.21.15.96:6789/0 conn(0x7f383c09a9d0 legacy=0x7f383c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.668+0000 7f38420b2640 1 -- 172.21.15.70:0/4248902344 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c070a90 con 0x7f383c15fb30 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.668+0000 7f38420b2640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c0d0d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3834009e80 tx=0x7f383402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee60b100628b38f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.668+0000 7f38408af640 1 -- 172.21.15.70:0/4248902344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383402caf0 con 0x7f383c15fb30 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.668+0000 7f38408af640 1 -- 172.21.15.70:0/4248902344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f383402cc90 con 0x7f383c15fb30 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.668+0000 7f38408af640 1 -- 172.21.15.70:0/4248902344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834034610 con 0x7f383c15fb30 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 -- 172.21.15.70:0/4248902344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 msgr2=0x7f383c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c0d0d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3834009e80 tx=0x7f383402bf50 comp rx=0 tx=0).stop 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 -- 172.21.15.70:0/4248902344 shutdown_connections 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c0d0d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 --2- 172.21.15.70:0/4248902344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0701f0 0x7f383c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 -- 172.21.15.70:0/4248902344 >> 172.21.15.70:0/4248902344 conn(0x7f383c0d2460 msgr2=0x7f383c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 -- 172.21.15.70:0/4248902344 shutdown_connections 2024-04-15T15:15:36.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.669+0000 7f3843b15640 1 -- 172.21.15.70:0/4248902344 wait complete. 2024-04-15T15:15:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.670+0000 7f3843b15640 1 Processor -- start 2024-04-15T15:15:36.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.670+0000 7f3843b15640 1 -- start start 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0701f0 0x7f383c150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 0x7f383c0b5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15ffc0 con 0x7f383c15fb30 2024-04-15T15:15:36.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f383c071050 con 0x7f383c0701f0 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f3843b15640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f383c0b1c60 con 0x7f383c09a9d0 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50640/0 (socket says 172.21.15.70:50640) 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 -- 172.21.15.70:0/1220864489 learned_addr learned my addr 172.21.15.70:0/1220864489 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38410b0640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 0x7f383c0b5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38418b1640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0701f0 0x7f383c150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 -- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 msgr2=0x7f383c0b5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 0x7f383c0b5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 -- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0701f0 msgr2=0x7f383c150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0701f0 0x7f383c150850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38420b2640 1 -- 172.21.15.70:0/1220864489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c0cc320 con 0x7f383c15fb30 2024-04-15T15:15:36.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.671+0000 7f38410b0640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 0x7f383c0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f38420b2640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f383402bf90 tx=0x7f3834004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834044020 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3834040690 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c0b7ab0 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3834009510 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.672+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c0af530 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.673+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0689e0 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.673+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3834047020 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.674+0000 7f38327fc640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 0x7f382006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.674+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f38340ae910 con 0x7f383c15fb30 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.674+0000 7f38418b1640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 0x7f382006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:36.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.674+0000 7f38418b1640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 0x7f382006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3838007fd0 tx=0x7f3838009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:36.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.678+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f383404e030 con 0x7f383c15fb30 2024-04-15T15:15:36.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.888+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f383c0d3370 con 0x7f383c15fb30 2024-04-15T15:15:36.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.889+0000 7f38327fc640 1 -- 172.21.15.70:0/1220864489 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f383c0b7ab0 con 0x7f383c15fb30 2024-04-15T15:15:36.926 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:36.926 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 msgr2=0x7f382006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 0x7f382006edb0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3838007fd0 tx=0x7f3838009040 comp rx=0 tx=0).stop 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 msgr2=0x7f383c1f4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f383402bf90 tx=0x7f3834004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:36.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 shutdown_connections 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f382006c8f0 0x7f382006edb0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c15fb30 0x7f383c1f4fa0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c09a9d0 0x7f383c0b5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 --2- 172.21.15.70:0/1220864489 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0701f0 0x7f383c150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 >> 172.21.15.70:0/1220864489 conn(0x7f383c0d2460 msgr2=0x7f383c15e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 shutdown_connections 2024-04-15T15:15:36.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:36.891+0000 7f3843b15640 1 -- 172.21.15.70:0/1220864489 wait complete. 2024-04-15T15:15:37.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:15:09.639145+0000', 'modified': '2024-04-15T15:15:25.681548+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4293}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4293': {'gid': 4293, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.70:6837/3048652815', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6835', 'nonce': 3048652815}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3048652815}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4293, 'qdb_cluster': [4293]} 2024-04-15T15:15:37.332 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:15:37.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.435+0000 7ff6b879d640 1 Processor -- start 2024-04-15T15:15:37.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.435+0000 7ff6b879d640 1 -- start start 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b879d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6b00701f0 0x7ff6b0160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b879d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b0070720 con 0x7ff6b015fb30 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b879d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6b0070a90 con 0x7ff6b009a9d0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b879d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff6b009ba90 con 0x7ff6b00701f0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b5d38640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6b00701f0 0x7ff6b0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b5d38640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff6b00701f0 0x7ff6b0160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50648/0 (socket says 172.21.15.70:50648) 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b6d3a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff6b015fb30 0x7ff6b00d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43296/0 (socket says 172.21.15.70:43296) 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b5d38640 1 -- 172.21.15.70:0/4129930453 learned_addr learned my addr 172.21.15.70:0/4129930453 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.436+0000 7ff6b5d38640 1 -- 172.21.15.70:0/4129930453 >> v1:172.21.15.96:6789/0 conn(0x7ff6b009a9d0 legacy=0x7ff6b009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5d38640 1 -- 172.21.15.70:0/4129930453 >> v1:172.21.15.70:6789/0 conn(0x7ff6b015fb30 legacy=0x7ff6b00d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5d38640 1 -- 172.21.15.70:0/4129930453 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b0070d70 con 0x7ff6b00701f0 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5537640 1 -- 172.21.15.70:0/4129930453 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578485054 0 0) 0x7ff6b0070720 con 0x7ff6b015fb30 2024-04-15T15:15:37.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/4129930453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b00701f0 0x7ff6b0160420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00098a0 tx=0x7ff6a002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ce65eff7d7032f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5537640 1 -- 172.21.15.70:0/4129930453 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0034070 con 0x7ff6b00701f0 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5537640 1 -- 172.21.15.70:0/4129930453 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a0030840 con 0x7ff6b00701f0 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.437+0000 7ff6b5537640 1 -- 172.21.15.70:0/4129930453 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0030c10 con 0x7ff6b00701f0 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 -- 172.21.15.70:0/4129930453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b00701f0 msgr2=0x7ff6b0160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 --2- 172.21.15.70:0/4129930453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b00701f0 0x7ff6b0160420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00098a0 tx=0x7ff6a002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 -- 172.21.15.70:0/4129930453 shutdown_connections 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 --2- 172.21.15.70:0/4129930453 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b00701f0 0x7ff6b0160420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 -- 172.21.15.70:0/4129930453 >> 172.21.15.70:0/4129930453 conn(0x7ff6b00d2460 msgr2=0x7ff6b00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 -- 172.21.15.70:0/4129930453 shutdown_connections 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.438+0000 7ff6b879d640 1 -- 172.21.15.70:0/4129930453 wait complete. 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.439+0000 7ff6b879d640 1 Processor -- start 2024-04-15T15:15:37.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.439+0000 7ff6b879d640 1 -- start start 2024-04-15T15:15:37.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff6b00701f0 0x7ff6b014e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b015fb30 0x7ff6b014b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b015ffc0 con 0x7ff6b009a9d0 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b00d08c0 con 0x7ff6b015fb30 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b879d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b00b1c60 con 0x7ff6b00701f0 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50662/0 (socket says 172.21.15.70:50662) 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1105697808 learned_addr learned my addr 172.21.15.70:0/1105697808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b015fb30 0x7ff6b014b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff6b00701f0 0x7ff6b014e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff6b00701f0 msgr2=0x7ff6b014e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff6b00701f0 0x7ff6b014e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b015fb30 msgr2=0x7ff6b014b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b015fb30 0x7ff6b014b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1105697808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b00c6af0 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.440+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00098a0 tx=0x7ff6a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.441+0000 7ff6a6ffd640 1 -- 172.21.15.70:0/1105697808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0034040 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.441+0000 7ff6a6ffd640 1 -- 172.21.15.70:0/1105697808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a0039d90 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.441+0000 7ff6a6ffd640 1 -- 172.21.15.70:0/1105697808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0043610 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.441+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b00c6dd0 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.441+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b00c7390 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.442+0000 7ff6a6ffd640 1 -- 172.21.15.70:0/1105697808 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff6a004a060 con 0x7ff6b009a9d0 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.443+0000 7ff6a6ffd640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 0x7ff69406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.443+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 0x7ff69406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.443+0000 7ff6a6ffd640 1 -- 172.21.15.70:0/1105697808 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff6a00ae060 con 0x7ff6b009a9d0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.443+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 0x7ff69406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00a750 tx=0x7ff6ac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 Processor -- start 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 -- start start 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66962f0 0x556ce66c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556ce66009c0 con 0x556ce66c2e10 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556ce65a7650 con 0x556ce66962f0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.444+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x556ce65fc980 con 0x556ce66c31f0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50674/0 (socket says 172.21.15.70:50674) 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 -- 172.21.15.70:0/884920236 learned_addr learned my addr 172.21.15.70:0/884920236 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66962f0 0x556ce66c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 -- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 msgr2=0x556ce66c5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 -- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66962f0 msgr2=0x556ce66c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66962f0 0x556ce66c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 -- 172.21.15.70:0/884920236 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce63132d0 con 0x556ce66c2e10 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff6b6539640 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c960 tx=0x7ff6ac00f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f587da317e7f4d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff693fff640 1 -- 172.21.15.70:0/884920236 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac019040 con 0x556ce66c2e10 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff693fff640 1 -- 172.21.15.70:0/884920236 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac022070 con 0x556ce66c2e10 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.445+0000 7ff693fff640 1 -- 172.21.15.70:0/884920236 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac013dc0 con 0x556ce66c2e10 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- 172.21.15.70:0/884920236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 msgr2=0x556ce6695db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00c960 tx=0x7ff6ac00f000 comp rx=0 tx=0).stop 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- 172.21.15.70:0/884920236 shutdown_connections 2024-04-15T15:15:37.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66962f0 0x556ce66c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 --2- 172.21.15.70:0/884920236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce6695db0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- 172.21.15.70:0/884920236 >> 172.21.15.70:0/884920236 conn(0x556ce6698020 msgr2=0x556ce66e0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- 172.21.15.70:0/884920236 shutdown_connections 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- 172.21.15.70:0/884920236 wait complete. 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.446+0000 7ff6bb384740 1 -- start start 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.447+0000 7ff6bb384740 1 Processor -- start 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.447+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.447+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 0x556ce66c48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.447+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce66c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556ce669a5c0 con 0x556ce66c2e10 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556ce65baa10 con 0x556ce66c31f0 2024-04-15T15:15:37.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x556ce66999b0 con 0x556ce66bb1d0 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 10 client.0 _ll_register_callbacks cb 0x556ce65f6250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50684/0 (socket says 172.21.15.70:50684) 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 -- 172.21.15.70:0/2158443327 learned_addr learned my addr 172.21.15.70:0/2158443327 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce66c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 0x556ce66c48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 -- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 msgr2=0x556ce66c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce66c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 -- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 msgr2=0x556ce66c48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 0x556ce66c48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 -- 172.21.15.70:0/2158443327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce63dd6b0 con 0x556ce66c2e10 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce66c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 0x556ce66c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:37.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6b6539640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac0137d0 tx=0x7ff6ac004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79647a9b81cac836 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6917fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac02b070 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff6ac013980 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6ac022070 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7ff6ac0044d0 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 10 client.4854 fetch_fsmap learned FSMap version 8 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x556ce66c89b0 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.448+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6ac022070 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7ff6ac01c030 con 0x556ce66c2e10 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 10 client.4854 fetch_fsmap finished waiting for FSMap version 8 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 10 client.4854 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 20 client.4854 populate_metadata read hostname 'smithi070' 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66be930 0x556ce66ff510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.460 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 4 client.4854 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 --> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ce660e3e0 con 0x556ce66be930 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66be930 0x556ce66ff510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66be930 0x556ce66ff510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebb5101f499a49df server_cookie=1992b18af541cf10 in_seq=0 out_seq=0 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.449+0000 7ff6917fa640 10 client.4854 ms_handle_connect on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6917fa640 1 -- 172.21.15.70:0/2158443327 <== mds.0 v2:172.21.15.70:6835/3048652815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556ce660e3e0 con 0x556ce66be930 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6917fa640 10 client.4854 handle_command_reply: tid=0 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 client.4854 shutdown 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 msgr2=0x556ce66e1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac0137d0 tx=0x7ff6ac004680 comp rx=0 tx=0).stop 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 shutdown_connections 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66be930 0x556ce66ff510 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce66c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6917fa640 0 client.4854 ms_handle_reset on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:37.461 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c31f0 0x556ce66c48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2158443327 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66e1550 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 >> 172.21.15.70:0/2158443327 conn(0x556ce66962f0 msgr2=0x556ce66e40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 shutdown_connections 2024-04-15T15:15:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 1 -- 172.21.15.70:0/2158443327 wait complete. 2024-04-15T15:15:37.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.450+0000 7ff6bb384740 20 client.4854 trim_cache size 0 max 16384 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.468+0000 7ff6bb384740 1 Processor -- start 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.468+0000 7ff6bb384740 1 -- start start 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce6699910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c2e10 0x556ce66f0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b02012b0 con 0x556ce66c31f0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b02006a0 con 0x556ce66c2e10 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff6b0200c90 con 0x556ce66bb1d0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50698/0 (socket says 172.21.15.70:50698) 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 -- 172.21.15.70:0/2059608513 learned_addr learned my addr 172.21.15.70:0/2059608513 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 -- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 msgr2=0x556ce6699910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce6699910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 -- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c2e10 msgr2=0x556ce66f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c2e10 0x556ce66f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 -- 172.21.15.70:0/2059608513 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ce66bbcd0 con 0x556ce66c31f0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80070d0 tx=0x7ff6a8007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=456065c4e1f2460f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6937fe640 1 -- 172.21.15.70:0/2059608513 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80017d0 con 0x556ce66c31f0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6937fe640 1 -- 172.21.15.70:0/2059608513 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a8019070 con 0x556ce66c31f0 2024-04-15T15:15:37.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.469+0000 7ff6937fe640 1 -- 172.21.15.70:0/2059608513 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8001bd0 con 0x556ce66c31f0 2024-04-15T15:15:37.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- 172.21.15.70:0/2059608513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 msgr2=0x556ce66f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff6a80070d0 tx=0x7ff6a8007590 comp rx=0 tx=0).stop 2024-04-15T15:15:37.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- 172.21.15.70:0/2059608513 shutdown_connections 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c31f0 0x556ce66f2cd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66c2e10 0x556ce66f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 --2- 172.21.15.70:0/2059608513 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66bb1d0 0x556ce6699910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- 172.21.15.70:0/2059608513 >> 172.21.15.70:0/2059608513 conn(0x556ce66962f0 msgr2=0x556ce66913e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- 172.21.15.70:0/2059608513 shutdown_connections 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- 172.21.15.70:0/2059608513 wait complete. 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.470+0000 7ff6bb384740 1 -- start start 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 Processor -- start 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66bb1d0 0x556ce66c6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x556ce66c9690 con 0x556ce66c2e10 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x556ce66c5130 con 0x556ce66bb1d0 2024-04-15T15:15:37.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.471+0000 7ff6bb384740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x556ce66bc600 con 0x556ce66c31f0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6bb384740 10 client.0 _ll_register_callbacks cb 0x7ff6ac00c800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50702/0 (socket says 172.21.15.70:50702) 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1180659970 learned_addr learned my addr 172.21.15.70:0/1180659970 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b6d3a640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66bb1d0 0x556ce66c6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 msgr2=0x556ce66c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66bb1d0 msgr2=0x556ce66c6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66bb1d0 0x556ce66c6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 -- 172.21.15.70:0/1180659970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ce675b580 con 0x556ce66c2e10 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6b5d38640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00ac250 tx=0x7ff6a007bbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7aaec626e627bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6917fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:15:37.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a00846f0 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff6a0099e60 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6a8019070 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7ff6a00a2050 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6bb384740 10 client.4860 fetch_fsmap learned FSMap version 8 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x556ce66f35c0 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.472+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a8019070 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7ff6a00a5020 con 0x556ce66c2e10 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 10 client.4860 fetch_fsmap finished waiting for FSMap version 8 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 10 client.4860 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 20 client.4860 populate_metadata read hostname 'smithi070' 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66d6530 0x556ce66d6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 4 client.4860 mds_command: new command op to 4293 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 --> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] -- command(tid 0: {"prefix": "status"}) -- 0x556ce6608250 con 0x556ce66d6530 2024-04-15T15:15:37.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66d6530 0x556ce66d6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6917fa640 10 client.4860 ms_handle_connect on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6b6539640 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66d6530 0x556ce66d6910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbff063457eb912 server_cookie=30bc3e796a62f0e9 in_seq=0 out_seq=0 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6917fa640 1 -- 172.21.15.70:0/1180659970 <== mds.0 v2:172.21.15.70:6835/3048652815 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x556ce6608250 con 0x556ce66d6530 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6917fa640 10 client.4860 handle_command_reply: tid=0 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 client.4860 shutdown 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 msgr2=0x556ce66f2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00ac250 tx=0x7ff6a007bbd0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 shutdown_connections 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:6835/3048652815,v1:172.21.15.70:6837/3048652815] conn(0x556ce66d6530 0x556ce66d6910 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x556ce66c31f0 0x556ce66c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x556ce66c2e10 0x556ce66f2cd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6917fa640 0 client.4860 ms_handle_reset on v2:172.21.15.70:6835/3048652815 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 --2- 172.21.15.70:0/1180659970 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x556ce66bb1d0 0x556ce66c6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 >> 172.21.15.70:0/1180659970 conn(0x556ce66962f0 msgr2=0x556ce669bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 shutdown_connections 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.473+0000 7ff6bb384740 1 -- 172.21.15.70:0/1180659970 wait complete. 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6bb384740 20 client.4860 trim_cache size 0 max 16384 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "074678c8-a3ad-4fb8-89f1-2c7d457fbbe5", 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4293, 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-15T15:15:37.482 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 27.189510212999998, 2024-04-15T15:15:37.483 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 8, 2024-04-15T15:15:37.483 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 25, 2024-04-15T15:15:37.483 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 25, 2024-04-15T15:15:37.483 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 33.847959936999999 2024-04-15T15:15:37.483 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 msgr2=0x7ff69406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 0x7ff69406edb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac00a750 tx=0x7ff6ac008040 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 msgr2=0x7ff6b014b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff6a00098a0 tx=0x7ff6a00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 shutdown_connections 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff6b015fb30 0x7ff6b014b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff69406c8f0 0x7ff69406edb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6b009a9d0 0x7ff6b014b3f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 --2- 172.21.15.70:0/1105697808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff6b00701f0 0x7ff6b014e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 >> 172.21.15.70:0/1105697808 conn(0x7ff6b00d2460 msgr2=0x7ff6b0161b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 shutdown_connections 2024-04-15T15:15:37.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.474+0000 7ff6b879d640 1 -- 172.21.15.70:0/1105697808 wait complete. 2024-04-15T15:15:37.487 INFO:teuthology.run_tasks:Running task sequential... 2024-04-15T15:15:37.496 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-15T15:15:37.496 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-15T15:15:37.497 INFO:teuthology.task.print:Enabling mgr modules 2024-04-15T15:15:37.497 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-15T15:15:37.507 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-15T15:15:37.507 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-15T15:15:37.507 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-15T15:15:37.507 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:37.507 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:37.507 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-15T15:15:37.507 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:37.507 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:37.507 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-15T15:15:37.508 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:37.508 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:37.508 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-15T15:15:37.508 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-15T15:15:37.508 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:37.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:37.508 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:15:37.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:37.529 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link delete ceph-brx 2024-04-15T15:15:37.604 INFO:teuthology.orchestra.run.smithi096.stderr:Cannot find device "ceph-brx" 2024-04-15T15:15:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:15:37.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:37.606 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:15:37.622 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-15T15:15:37.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:37.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-15T15:15:37.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:37.691 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-15T15:15:37.787 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-15T15:15:37.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:37.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.890+0000 7fbb06cc8640 1 Processor -- start 2024-04-15T15:15:37.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.890+0000 7fbb06cc8640 1 -- start start 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb06cc8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb0006b9e0 0x7fbb0006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb06cc8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbb0015d270 0x7fbb0015d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb06cc8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb0006c300 con 0x7fbb000b3a30 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb06cc8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb000cd160 con 0x7fbb0006b9e0 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb06cc8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbb000cd440 con 0x7fbb0015d270 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb0006b9e0 0x7fbb0006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb0006b9e0 0x7fbb0006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50720/0 (socket says 172.21.15.70:50720) 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 -- 172.21.15.70:0/2536881326 learned_addr learned my addr 172.21.15.70:0/2536881326 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb05265640 1 --2- 172.21.15.70:0/2536881326 >> v2:172.21.15.96:3300/0 conn(0x7fbb0015d270 0x7fbb0015d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 -- 172.21.15.70:0/2536881326 >> v2:172.21.15.96:3300/0 conn(0x7fbb0015d270 msgr2=0x7fbb0015d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2536881326 >> v2:172.21.15.96:3300/0 conn(0x7fbb0015d270 0x7fbb0015d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 -- 172.21.15.70:0/2536881326 >> v1:172.21.15.96:6789/0 conn(0x7fbb000b3a30 legacy=0x7fbb0015ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:37.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.891+0000 7fbb04a64640 1 -- 172.21.15.70:0/2536881326 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb000cd720 con 0x7fbb0006b9e0 2024-04-15T15:15:37.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.892+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2536881326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0006b9e0 0x7fbb0006bdc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbb00073f40 tx=0x7fbaf402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec7618de48e08edb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.892+0000 7fbaff7fe640 1 -- 172.21.15.70:0/2536881326 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4035020 con 0x7fbb0006b9e0 2024-04-15T15:15:37.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.892+0000 7fbaff7fe640 1 -- 172.21.15.70:0/2536881326 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf4031870 con 0x7fbb0006b9e0 2024-04-15T15:15:37.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.892+0000 7fbaff7fe640 1 -- 172.21.15.70:0/2536881326 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4031c40 con 0x7fbb0006b9e0 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2536881326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0006b9e0 msgr2=0x7fbb0006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2536881326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0006b9e0 0x7fbb0006bdc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbb00073f40 tx=0x7fbaf402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2536881326 shutdown_connections 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2536881326 >> v2:172.21.15.96:3300/0 conn(0x7fbb0015d270 0x7fbb0015d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2536881326 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0006b9e0 0x7fbb0006bdc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2536881326 >> 172.21.15.70:0/2536881326 conn(0x7fbb0009a9d0 msgr2=0x7fbb000b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2536881326 shutdown_connections 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.893+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2536881326 wait complete. 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.894+0000 7fbb06cc8640 1 Processor -- start 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.894+0000 7fbb06cc8640 1 -- start start 2024-04-15T15:15:37.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 0x7fbb0014e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 0x7fbb000b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb000cce50 con 0x7fbb000b3a30 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb000cdce0 con 0x7fbb0015d270 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb06cc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbb000ce200 con 0x7fbb0006b9e0 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50722/0 (socket says 172.21.15.70:50722) 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 -- 172.21.15.70:0/2558903891 learned_addr learned my addr 172.21.15.70:0/2558903891 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 0x7fbb0014e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb05265640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 0x7fbb000b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 -- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 msgr2=0x7fbb0014e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 0x7fbb0014e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 -- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 msgr2=0x7fbb000b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 0x7fbb000b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:37.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 -- 172.21.15.70:0/2558903891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb000b4140 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 0x7fbb0014e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbb05265640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 0x7fbb000b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafffff640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c460 tx=0x7fbaec00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.895+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec016020 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.896+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec012870 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.896+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb000b4420 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.896+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec012c60 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.896+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb000c53d0 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.897+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb000690f0 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.897+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbaec019020 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.897+0000 7fbafd7fa640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 0x7fbae406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.898+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbaec08bbb0 con 0x7fbb000b3a30 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.898+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 0x7fbae406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:37.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.898+0000 7fbb04a64640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 0x7fbae406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:37.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:37.902+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbaec020030 con 0x7fbb000b3a30 2024-04-15T15:15:38.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.115+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbb00073200 con 0x7fbb000b3a30 2024-04-15T15:15:38.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.115+0000 7fbafd7fa640 1 -- 172.21.15.70:0/2558903891 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7fbaec059ca0 con 0x7fbb000b3a30 2024-04-15T15:15:38.117 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 msgr2=0x7fbae406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 0x7fbae406edb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).stop 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 msgr2=0x7fbb000b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbaec00c460 tx=0x7fbaec00c920 comp rx=0 tx=0).stop 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 shutdown_connections 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0015d270 0x7fbb000b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c8f0 0x7fbae406edb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb000b3a30 0x7fbb000b2ea0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 --2- 172.21.15.70:0/2558903891 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0006b9e0 0x7fbb0014e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:38.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 >> 172.21.15.70:0/2558903891 conn(0x7fbb0009a9d0 msgr2=0x7fbb000b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:38.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 shutdown_connections 2024-04-15T15:15:38.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38.117+0000 7fbb06cc8640 1 -- 172.21.15.70:0/2558903891 wait complete. 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:38.130 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:38.131 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:38.131 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:15:38.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:38.131 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:15:38.147 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:38.147 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:38.147 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:38.148 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:38.149 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:38.149 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42713sec preferred_lft 42713sec 2024-04-15T15:15:38.149 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:15:38.149 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:38.150 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set ceph-brx up 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:38.150 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:38.236 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:38.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:38.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:38.301 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:38.327 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:15:38.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:38.329 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:15:38.387 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:15:38.387 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:38.387 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:38.387 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:38.387 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:38.387 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:38.388 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:38.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:38.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:38.388 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:38.388 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:38.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:38.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:38.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:38.527 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:15:38.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:38.543 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-15T15:15:38.599 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:38.599 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:38.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:38.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-15T15:15:38.599 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:38.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:38.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:38.722 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-15T15:15:38.723 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:38.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:38.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:38.855 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:38.855 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:38.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-15T15:15:38.855 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:38.855 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:38.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:38.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:38.940 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:15:38.940 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:38.940 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:38.957 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:15:38.957 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:38.957 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:39.012 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:39.082 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:39.139 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:39.139 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:39.139 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:39.139 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:39.140 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:39.141 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:39.142 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:39.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:39.143 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:39.195 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:39.195 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:15:39.239 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:39.267 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:39.323 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:39.324 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:39.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:39.325 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:39.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-15T15:15:39.324+0000 7f0ca5a93480 -1 init, newargv = 0x55fcb7f41090 newargc=15 2024-04-15T15:15:39.326 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[47246]: starting ceph client 2024-04-15T15:15:39.330 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[47246]: starting fuse 2024-04-15T15:15:39.343 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:15:39.344 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:15:39.345 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-15T15:15:39.346 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:15:39.466 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.47246.asok 2024-04-15T15:15:39.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:39.476 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47246.asok 2024-04-15T15:15:39.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:39.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47246.asok status 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:15:39.613 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "47246", 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4872, 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4872 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:39.614 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3253368493 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3253368493 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4872 192.168.0.1:0/3253368493", 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/3253368493", 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 8, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 25, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:39.615 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:15:39.616 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:15:39.616 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:15:39.621 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:39.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.722+0000 7f0d26f75640 1 Processor -- start 2024-04-15T15:15:39.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.722+0000 7f0d26f75640 1 -- start start 2024-04-15T15:15:39.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d26f75640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0d2015eee0 0x7f0d200700d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d26f75640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d200ce6d0 con 0x7f0d200cff90 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d26f75640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d200ce9b0 con 0x7f0d20070d10 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d26f75640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0d2006fa70 con 0x7f0d2015eee0 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d1ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0d200cff90 0x7f0d200cbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43298/0 (socket says 172.21.15.70:43298) 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d1ffff640 1 -- 172.21.15.70:0/1018271411 learned_addr learned my addr 172.21.15.70:0/1018271411 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d24d11640 1 --2- 172.21.15.70:0/1018271411 >> v2:172.21.15.96:3300/0 conn(0x7f0d2015eee0 0x7f0d200700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108437370 0 0) 0x7f0d200ce6d0 con 0x7f0d200cff90 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.723+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d10003520 con 0x7f0d200cff90 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3273648144 0 0) 0x7f0d10003520 con 0x7f0d200cff90 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 >> v2:172.21.15.96:3300/0 conn(0x7f0d2015eee0 msgr2=0x7f0d200700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 --2- 172.21.15.70:0/1018271411 >> v2:172.21.15.96:3300/0 conn(0x7f0d2015eee0 0x7f0d200700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 >> v1:172.21.15.96:6789/0 conn(0x7f0d20070d10 legacy=0x7f0d200b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2006fd50 con 0x7f0d200cff90 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f0d0c001670 con 0x7f0d200cff90 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 >> v1:172.21.15.70:6789/0 conn(0x7f0d200cff90 legacy=0x7f0d200cbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 0x7f0d10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d10004800 0x7f0d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100051c0 0x7f0d1000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2006fa70 con 0x7f0d100039b0 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d1000c060 con 0x7f0d10004800 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1000c260 con 0x7f0d100051c0 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 0x7f0d10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d1ffff640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d10004800 0x7f0d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d25512640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100051c0 0x7f0d1000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 -- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100051c0 msgr2=0x7f0d1000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100051c0 0x7f0d1000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 -- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d10004800 msgr2=0x7f0d10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d10004800 0x7f0d10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d24d11640 1 -- 172.21.15.70:0/1018271411 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d10003520 con 0x7f0d100039b0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.724+0000 7f0d25512640 1 -- 172.21.15.70:0/1018271411 reap_dead start 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d24d11640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 0x7f0d10003dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009ac0 tx=0x7f0d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e157e99f2013017 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14039070 con 0x7f0d100039b0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d1402c6d0 con 0x7f0d100039b0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d1f7fe640 1 -- 172.21.15.70:0/1018271411 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d14009690 con 0x7f0d100039b0 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 -- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 msgr2=0x7f0d10003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 0x7f0d10003dd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0d14009ac0 tx=0x7f0d10005830 comp rx=0 tx=0).stop 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 -- 172.21.15.70:0/1018271411 shutdown_connections 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 --2- 172.21.15.70:0/1018271411 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d100039b0 0x7f0d10003dd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 -- 172.21.15.70:0/1018271411 >> 172.21.15.70:0/1018271411 conn(0x7f0d2009a9d0 msgr2=0x7f0d200b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.725+0000 7f0d26f75640 1 -- 172.21.15.70:0/1018271411 shutdown_connections 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.726+0000 7f0d26f75640 1 -- 172.21.15.70:0/1018271411 wait complete. 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.727+0000 7f0d26f75640 1 Processor -- start 2024-04-15T15:15:39.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.727+0000 7f0d26f75640 1 -- start start 2024-04-15T15:15:39.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.727+0000 7f0d26f75640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100039b0 0x7f0d201f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.727+0000 7f0d26f75640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.727+0000 7f0d26f75640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d201f4c60 0x7f0d200c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d26f75640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0d20070250 con 0x7f0d200c4df0 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d26f75640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0d200700d0 con 0x7f0d201f4c60 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d26f75640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0d200703d0 con 0x7f0d100039b0 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50738/0 (socket says 172.21.15.70:50738) 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 -- 172.21.15.70:0/26138298 learned_addr learned my addr 172.21.15.70:0/26138298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d25512640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d201f4c60 0x7f0d200c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 -- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100039b0 msgr2=0x7f0d201f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d24d11640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100039b0 0x7f0d201f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100039b0 0x7f0d201f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 -- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d201f4c60 msgr2=0x7f0d200c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d201f4c60 0x7f0d200c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 -- 172.21.15.70:0/26138298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d201f8270 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1ffff640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0072c0 tx=0x7f0d0c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00f8e0 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d0c018070 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d201f8550 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c00fd00 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.728+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d201f4430 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.729+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d200690f0 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.730+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0d0c01d460 con 0x7f0d200c4df0 2024-04-15T15:15:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.730+0000 7f0d1d7fa640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 0x7f0d0406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:39.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.730+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0d0c08bf80 con 0x7f0d200c4df0 2024-04-15T15:15:39.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.730+0000 7f0d24d11640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 0x7f0d0406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:39.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.730+0000 7f0d24d11640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 0x7f0d0406ede0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d14036000 tx=0x7f0d14030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:39.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.734+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0d0c05a070 con 0x7f0d200c4df0 2024-04-15T15:15:39.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.947+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f0d200b74a0 con 0x7f0d200c4df0 2024-04-15T15:15:39.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.947+0000 7f0d1d7fa640 1 -- 172.21.15.70:0/26138298 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f0d0c059970 con 0x7f0d200c4df0 2024-04-15T15:15:39.949 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 msgr2=0x7f0d0406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 0x7f0d0406ede0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0d14036000 tx=0x7f0d14030000 comp rx=0 tx=0).stop 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 msgr2=0x7f0d2014f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c0072c0 tx=0x7f0d0c007780 comp rx=0 tx=0).stop 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 shutdown_connections 2024-04-15T15:15:39.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0d201f4c60 0x7f0d200c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0d0406c920 0x7f0d0406ede0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0d200c4df0 0x7f0d2014f8f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 --2- 172.21.15.70:0/26138298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0d100039b0 0x7f0d201f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 >> 172.21.15.70:0/26138298 conn(0x7f0d2009a9d0 msgr2=0x7f0d200b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 shutdown_connections 2024-04-15T15:15:39.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:39.949+0000 7f0d26f75640 1 -- 172.21.15.70:0/26138298 wait complete. 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:39.962 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:15:39.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:39.963 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:15:39.979 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42711sec preferred_lft 42711sec 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 82:8a:03:3a:60:03 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:39.980 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c488:39ff:fe35:ded1/64 scope link tentative 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 82:8a:03:3a:60:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::808a:3ff:fe3a:6003/64 scope link tentative 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:39.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:39.981 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:40.047 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:15:40.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.048 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:15:40.105 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:15:40.106 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:40.106 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:40.106 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:40.106 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:40.106 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:40.106 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:40.106 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:40.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:40.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:40.107 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:40.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:40.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:40.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.244 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:15:40.259 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:15:40.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.260 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list-id 2024-04-15T15:15:40.315 INFO:teuthology.orchestra.run.smithi070.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-15T15:15:40.316 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:40.316 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:40.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:40.316 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-15T15:15:40.316 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:40.404 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:40.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:40.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:15:40.467 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether ae:6d:eb:16:6b:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ac6d:ebff:fe16:6bfe/64 scope link tentative 2024-04-15T15:15:40.468 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.470 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.538 INFO:teuthology.orchestra.run.smithi070.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether ae:6d:eb:16:6b:fe brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ac6d:ebff:fe16:6bfe/64 scope link tentative 2024-04-15T15:15:40.539 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:40.540 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-15T15:15:40.540 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-15T15:15:40.541 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:40.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:40.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.1 up 2024-04-15T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:15:40.709 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:15:40.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:40.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:40.793 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:15:40.793 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:40.793 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:40.808 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:15:40.809 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:40.809 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:40.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:40.936 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:40.991 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:40.991 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:40.992 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:40.993 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:40.994 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:40.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:40.995 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:41.051 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:15:41.051 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:15:41.051 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-15T15:15:41.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:15:41.122 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:15:41.179 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:41.180 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:41.181 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:15:41.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:41.182 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:15:41.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:2024-04-15T15:15:41.180+0000 7fd0e3a36480 -1 init, newargv = 0x560d53d17bd0 newargc=15 2024-04-15T15:15:41.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[47744]: starting ceph client 2024-04-15T15:15:41.186 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[47744]: starting fuse 2024-04-15T15:15:41.198 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:15:41.198 INFO:teuthology.orchestra.run.smithi070.stdout:44 2024-04-15T15:15:41.198 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:41.198 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:15:41.199 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:41.200 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:15:41.201 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.2")) 2024-04-15T15:15:41.201 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:15:41.320 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.2.47744.asok 2024-04-15T15:15:41.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:41.333 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.47744.asok 2024-04-15T15:15:41.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:41.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.47744.asok status 2024-04-15T15:15:41.471 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "2", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "47744", 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4887, 2024-04-15T15:15:41.472 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 4887 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3643783223 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 3643783223 2024-04-15T15:15:41.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.4887 192.168.0.2:0/3643783223", 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.2:0/3643783223", 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 8, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 25, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:15:41.474 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:15:41.479 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:41.485 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:41.525 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:15:41.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.583+0000 7f2eaafbd640 1 Processor -- start 2024-04-15T15:15:41.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.583+0000 7f2eaafbd640 1 -- start start 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2eaafbd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2ea409ba10 0x7f2ea409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2eaafbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea40d0490 con 0x7f2ea40b2a30 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2eaafbd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ea40b1990 con 0x7f2ea40d0940 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2eaafbd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2ea40b1c70 con 0x7f2ea409ba10 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2ea409ba10 0x7f2ea409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2ea409ba10 0x7f2ea409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50742/0 (socket says 172.21.15.70:50742) 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2ea955a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2ea40b2a30 0x7f2ea40b2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:43306/0 (socket says 172.21.15.70:43306) 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 -- 172.21.15.70:0/3394151114 learned_addr learned my addr 172.21.15.70:0/3394151114 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:41.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 -- 172.21.15.70:0/3394151114 >> v1:172.21.15.96:6789/0 conn(0x7f2ea40d0940 legacy=0x7f2ea40d0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 -- 172.21.15.70:0/3394151114 >> v1:172.21.15.70:6789/0 conn(0x7f2ea40b2a30 legacy=0x7f2ea40b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.584+0000 7f2e9bfff640 1 -- 172.21.15.70:0/3394151114 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea40b1f50 con 0x7f2ea409ba10 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.585+0000 7f2e9bfff640 1 --2- 172.21.15.70:0/3394151114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea409b3e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009870 tx=0x7f2e9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9325ab3bcf8595d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.585+0000 7f2e9b7fe640 1 -- 172.21.15.70:0/3394151114 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94035020 con 0x7f2ea409ba10 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.585+0000 7f2e9b7fe640 1 -- 172.21.15.70:0/3394151114 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e94031870 con 0x7f2ea409ba10 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.585+0000 7f2e9b7fe640 1 -- 172.21.15.70:0/3394151114 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e94031c40 con 0x7f2ea409ba10 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 -- 172.21.15.70:0/3394151114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 msgr2=0x7f2ea409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/3394151114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea409b3e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e94009870 tx=0x7f2e9402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 -- 172.21.15.70:0/3394151114 shutdown_connections 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/3394151114 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea409b3e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 -- 172.21.15.70:0/3394151114 >> 172.21.15.70:0/3394151114 conn(0x7f2ea40b3e50 msgr2=0x7f2ea40b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 -- 172.21.15.70:0/3394151114 shutdown_connections 2024-04-15T15:15:41.589 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.586+0000 7f2eaafbd640 1 -- 172.21.15.70:0/3394151114 wait complete. 2024-04-15T15:15:41.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.587+0000 7f2eaafbd640 1 Processor -- start 2024-04-15T15:15:41.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.587+0000 7f2eaafbd640 1 -- start start 2024-04-15T15:15:41.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.591 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2ea40b2a30 0x7f2ea415d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ea40d0940 0x7f2ea414d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea415ca30 con 0x7f2ea409ba10 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea40b2510 con 0x7f2ea40d0940 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea415cbb0 con 0x7f2ea40b2a30 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50756/0 (socket says 172.21.15.70:50756) 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 -- 172.21.15.70:0/1157427167 learned_addr learned my addr 172.21.15.70:0/1157427167 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea955a640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ea40d0940 0x7f2ea414d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2e9bfff640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2ea40b2a30 0x7f2ea415d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 -- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2ea40b2a30 msgr2=0x7f2ea415d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2ea40b2a30 0x7f2ea415d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 -- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ea40d0940 msgr2=0x7f2ea414d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ea40d0940 0x7f2ea414d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 -- 172.21.15.70:0/1157427167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea40c5f30 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2ea8d59640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c3f0 tx=0x7f2ea000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011790 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea0017070 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ea415e0e0 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0011b90 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.588+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ea415e6a0 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.590+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ea40689e0 con 0x7f2ea409ba10 2024-04-15T15:15:41.593 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.590+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ea001c460 con 0x7f2ea409ba10 2024-04-15T15:15:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.590+0000 7f2e997fa640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 0x7f2e9006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.590+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2ea008bda0 con 0x7f2ea409ba10 2024-04-15T15:15:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.590+0000 7f2e9bfff640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 0x7f2e9006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:41.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.591+0000 7f2e9bfff640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 0x7f2e9006ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e94036000 tx=0x7f2e9403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:41.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.595+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2ea0059d80 con 0x7f2ea409ba10 2024-04-15T15:15:41.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.808+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2ea4075350 con 0x7f2ea409ba10 2024-04-15T15:15:41.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.808+0000 7f2e997fa640 1 -- 172.21.15.70:0/1157427167 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f2ea0059680 con 0x7f2ea409ba10 2024-04-15T15:15:41.810 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.810+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 msgr2=0x7f2e9006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.810+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 0x7f2e9006ede0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2e94036000 tx=0x7f2e9403c040 comp rx=0 tx=0).stop 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.810+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 msgr2=0x7f2ea41f6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.810+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2ea000c3f0 tx=0x7f2ea000c8b0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 shutdown_connections 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2ea40d0940 0x7f2ea414d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2ea40b2a30 0x7f2ea415d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e9006c920 0x7f2e9006ede0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 --2- 172.21.15.70:0/1157427167 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2ea409ba10 0x7f2ea41f6050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 >> 172.21.15.70:0/1157427167 conn(0x7f2ea40b3e50 msgr2=0x7f2ea415a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 shutdown_connections 2024-04-15T15:15:41.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:41.811+0000 7f2eaafbd640 1 -- 172.21.15.70:0/1157427167 wait complete. 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:41.823 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:41.824 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:41.824 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:15:41.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:41.824 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:41.844 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:41.845 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:41.846 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:41.846 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42684sec preferred_lft 42684sec 2024-04-15T15:15:41.846 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-04-15T15:15:41.846 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:41.846 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add name ceph-brx type bridge 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr flush dev ceph-brx 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set ceph-brx up 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-15T15:15:41.847 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:41.941 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:41 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:42.058 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:42.064 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:42.064 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:42.094 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:15:42.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:42.096 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:42.170 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:42.170 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:42.171 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:42.171 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:42.171 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:42.171 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:42.171 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:42.258 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:42.474 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:42.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:42.480 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:15:42.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:42.496 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list-id 2024-04-15T15:15:42.553 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:42.553 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:42.553 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:42.553 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-15T15:15:42.553 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:42.641 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:42.671 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:42.677 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-15T15:15:42.678 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:42.727 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:42.815 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:42.821 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:42.821 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:42.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-04-15T15:15:42.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set dev brx.0 master ceph-brx 2024-04-15T15:15:42.821 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:42.870 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:42.899 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:42.905 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:15:42.906 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:42.906 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:42.921 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:15:42.921 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:42.921 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:42.977 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:15:43.047 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:43.103 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:15:43.103 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:15:43.104 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:15:43.105 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:43.106 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:43.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:43.106 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:15:43.159 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:15:43.159 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:15:43.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:15:43.230 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:15:43.288 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:43.289 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:43.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:43.291 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:15:43.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:2024-04-15T15:15:43.288+0000 7f3bf9441480 -1 init, newargv = 0x5614b8948d00 newargc=15 2024-04-15T15:15:43.291 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[43988]: starting ceph client 2024-04-15T15:15:43.295 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[43988]: starting fuse 2024-04-15T15:15:43.306 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:15:43.307 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:15:43.307 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:43.308 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:43.309 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.1")) 2024-04-15T15:15:43.309 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-15T15:15:43.429 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.1.43988.asok 2024-04-15T15:15:43.432 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:43.440 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43988.asok 2024-04-15T15:15:43.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:43.440 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43988.asok status 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "1", 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-15T15:15:43.578 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "43988", 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4574, 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 4574 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:15:43.579 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3219185216 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3219185216 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.4574 192.168.0.1:0/3219185216", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/3219185216", 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 8, 2024-04-15T15:15:43.580 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 25, 2024-04-15T15:15:43.581 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:43.581 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-15T15:15:43.581 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-15T15:15:43.581 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-15T15:15:43.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:15:43.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.700+0000 7f91ab84e640 1 Processor -- start 2024-04-15T15:15:43.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.700+0000 7f91ab84e640 1 -- start start 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91ab84e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a40d04c0 0x7f91a4140380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91ab84e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a40cf1c0 con 0x7f91a406fcd0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91ab84e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a40cf440 con 0x7f91a40b6ac0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91ab84e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f91a40cf700 con 0x7f91a40d04c0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91a95ea640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f91a406fcd0 0x7f91a415bd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55852/0 (socket says 172.21.15.70:55852) 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91a95ea640 1 -- 172.21.15.70:0/523871804 learned_addr learned my addr 172.21.15.70:0/523871804 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91a8de9640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a40d04c0 0x7f91a4140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272201386 0 0) 0x7f91a0001110 con 0x7f91a406fcd0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9190003520 con 0x7f91a406fcd0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2409036661 0 0) 0x7f91a40cf440 con 0x7f91a40b6ac0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f91a0001110 con 0x7f91a40b6ac0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1683390089 0 0) 0x7f9190003520 con 0x7f91a406fcd0 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a40d04c0 msgr2=0x7f91a4140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a40d04c0 0x7f91a4140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 >> v1:172.21.15.96:6789/0 conn(0x7f91a40b6ac0 legacy=0x7f91a406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a40cf9e0 con 0x7f91a406fcd0 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f91a8de9640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a40d04c0 0x7f91a4140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.701+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f91a0001840 con 0x7f91a406fcd0 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 >> v1:172.21.15.70:6789/0 conn(0x7f91a406fcd0 legacy=0x7f91a415bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9190003b70 0x7f9190003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9190004800 0x7f9190004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91900051c0 0x7f919000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40cf700 con 0x7f91900051c0 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f919000bf00 con 0x7f9190004800 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f919000c100 con 0x7f9190003b70 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9190003b70 0x7f9190003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a9deb640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91900051c0 0x7f919000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a8de9640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9190004800 0x7f9190004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 -- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9190004800 msgr2=0x7f9190004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9190004800 0x7f9190004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 -- 172.21.15.70:0/523871804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91900051c0 msgr2=0x7f919000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91900051c0 0x7f919000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a95ea640 1 -- 172.21.15.70:0/523871804 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9190003520 con 0x7f9190003b70 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.702+0000 7f91a9deb640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91900051c0 0x7f919000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.703+0000 7f91a95ea640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9190003b70 0x7f9190003f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c4f0 tx=0x7f9190005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f003153ef4c8f52 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.703+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a000c0e0 con 0x7f9190003b70 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.703+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a0030070 con 0x7f9190003b70 2024-04-15T15:15:43.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.703+0000 7f919bfff640 1 -- 172.21.15.70:0/523871804 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a00058d0 con 0x7f9190003b70 2024-04-15T15:15:43.710 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 -- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9190003b70 msgr2=0x7f9190003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 --2- 172.21.15.70:0/523871804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9190003b70 0x7f9190003f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c4f0 tx=0x7f9190005830 comp rx=0 tx=0).stop 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91a9deb640 1 -- 172.21.15.70:0/523871804 reap_dead start 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 -- 172.21.15.70:0/523871804 shutdown_connections 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 -- 172.21.15.70:0/523871804 >> 172.21.15.70:0/523871804 conn(0x7f91a4073380 msgr2=0x7f91a406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 -- 172.21.15.70:0/523871804 shutdown_connections 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.704+0000 7f91ab84e640 1 -- 172.21.15.70:0/523871804 wait complete. 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.705+0000 7f91ab84e640 1 Processor -- start 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.705+0000 7f91ab84e640 1 -- start start 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91a414f350 0x7f91a40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a414e290 0x7f91a413e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40700b0 con 0x7f9190003b70 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f91a40cffa0 con 0x7f91a414f350 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f91a4070230 con 0x7f91a414e290 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45256/0 (socket says 172.21.15.70:45256) 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 -- 172.21.15.70:0/1539061511 learned_addr learned my addr 172.21.15.70:0/1539061511 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a8de9640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91a414f350 0x7f91a40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a9deb640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a414e290 0x7f91a413e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 -- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a414e290 msgr2=0x7f91a413e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a414e290 0x7f91a413e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 -- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91a414f350 msgr2=0x7f91a40b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91a414f350 0x7f91a40b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 -- 172.21.15.70:0/1539061511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a414f7b0 con 0x7f9190003b70 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91a95ea640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c4f0 tx=0x7f91a40cfbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0004bf0 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a0004d90 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a414d210 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.706+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0007290 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.707+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a414d7b0 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.708+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a40698a0 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.708+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f91a0048020 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.708+0000 7f9199ffb640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 0x7f918806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.709+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f91a00ad340 con 0x7f9190003b70 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.709+0000 7f91a8de9640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 0x7f918806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:43.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.709+0000 7f91a8de9640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 0x7f918806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f91a40b1cf0 tx=0x7f9194008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:43.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.713+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f91a40698a0 con 0x7f9190003b70 2024-04-15T15:15:43.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.930+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f91a40689e0 con 0x7f9190003b70 2024-04-15T15:15:43.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.931+0000 7f9199ffb640 1 -- 172.21.15.70:0/1539061511 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f91a40cf1c0 con 0x7f9190003b70 2024-04-15T15:15:43.933 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 msgr2=0x7f918806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 0x7f918806edb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f91a40b1cf0 tx=0x7f9194008040 comp rx=0 tx=0).stop 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 msgr2=0x7f91a4140eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f91a000c4f0 tx=0x7f91a40cfbb0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 shutdown_connections 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91a414e290 0x7f91a413e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f91a414f350 0x7f91a40b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f918806c8f0 0x7f918806edb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 --2- 172.21.15.70:0/1539061511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9190003b70 0x7f91a4140eb0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:43.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 >> 172.21.15.70:0/1539061511 conn(0x7f91a4073380 msgr2=0x7f91a40b1f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:43.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 shutdown_connections 2024-04-15T15:15:43.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:43.933+0000 7f91ab84e640 1 -- 172.21.15.70:0/1539061511 wait complete. 2024-04-15T15:15:43.945 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:15:43.946 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:15:43.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:43.946 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:43.963 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42681sec preferred_lft 42681sec 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether f6:ed:c4:10:c9:d9 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:15:43.964 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::c27:e4ff:fe37:c79c/64 scope link 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether f6:ed:c4:10:c9:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::f4ed:c4ff:fe10:c9d9/64 scope link tentative 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:43.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:43.965 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:15:44.033 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:15:44.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.035 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-15T15:15:44.092 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-15T15:15:44.092 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:15:44.092 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:15:44.092 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:15:44.092 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:15:44.092 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:44.092 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:44.092 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:15:44.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:15:44.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:15:44.093 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:44.180 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:44.224 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:44.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.230 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:15:44.246 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:15:44.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.247 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list-id 2024-04-15T15:15:44.302 INFO:teuthology.orchestra.run.smithi096.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-15T15:15:44.302 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:44.302 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:44.302 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:44.302 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-15T15:15:44.302 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:44.391 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:44.421 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:44.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.427 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ee:7d:e2:bd:de:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec7d:e2ff:febd:de9e/64 scope link tentative 2024-04-15T15:15:44.456 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether ee:7d:e2:bd:de:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-15T15:15:44.526 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.527 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec7d:e2ff:febd:de9e/64 scope link tentative 2024-04-15T15:15:44.527 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:15:44.528 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-15T15:15:44.528 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:44.617 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:44.692 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:44.698 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:15:44.698 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:15:44.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.1 up 2024-04-15T15:15:44.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set dev brx.1 master ceph-brx 2024-04-15T15:15:44.698 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:15:44.746 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:15:44.776 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:44.783 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:15:44.783 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:44.783 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:44.810 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:15:44.810 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:15:44.810 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:44.865 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:15:44.935 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:44.991 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:15:44.992 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:15:44.993 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:44.994 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:44.995 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:44.995 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:44.995 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:44.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:44.995 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:15:45.047 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:15:45.047 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-04-15T15:15:45.048 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-15T15:15:45.090 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:15:45.118 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:15:45.175 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:15:45.176 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:15:45.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.178 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:15:45.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:2024-04-15T15:15:45.176+0000 7f1fd620d480 -1 init, newargv = 0x55ae1f12eda0 newargc=15 2024-04-15T15:15:45.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:ceph-fuse[44444]: starting ceph client 2024-04-15T15:15:45.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:ceph-fuse[44444]: starting fuse 2024-04-15T15:15:45.194 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:15:45.194 INFO:teuthology.orchestra.run.smithi096.stdout:44 2024-04-15T15:15:45.194 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-04-15T15:15:45.194 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:15:45.194 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-15T15:15:45.194 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-15T15:15:45.194 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:15:45.195 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.3")) 2024-04-15T15:15:45.196 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-15T15:15:45.316 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.3.44444.asok 2024-04-15T15:15:45.319 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:15:45 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:15:45.326 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44444.asok 2024-04-15T15:15:45.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.326 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44444.asok status 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "3", 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-15T15:15:45.466 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "44444", 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 4586, 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 4586 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:15:45.467 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2680797015 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2680797015 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.4586 192.168.0.2:0/2680797015", 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.2:0/2680797015", 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 8, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 25, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:15:45.468 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-15T15:15:45.469 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-15T15:15:45.469 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-15T15:15:45.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.475 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-15T15:15:45.475 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:45.495 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-15T15:15:45.496 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:45.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:45.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.569 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-15T15:15:45.569 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:45.629 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-15T15:15:45.629 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:45.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:15:45.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.702 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-04-15T15:15:45.702 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:45.719 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-04-15T15:15:45.719 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:45.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.720 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:15:45.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.789 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-04-15T15:15:45.789 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:45.846 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-04-15T15:15:45.846 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:45.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:45.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:15:45.916 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-15T15:15:45.939 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.051+0000 7f2713b31640 1 Processor -- start 2024-04-15T15:15:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 -- start start 2024-04-15T15:15:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 0x7f270c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f270c15eae0 0x7f270c0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f270c0b47a0 con 0x7f270c15e700 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f270c0b53b0 con 0x7f270c15eae0 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.052+0000 7f2713b31640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f270c0b4a80 con 0x7f270c0d2360 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27110cc640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 0x7f270c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27120ce640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f270c15eae0 0x7f270c0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27118cd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f270c15e700 0x7f270c0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:36652/0 (socket says 172.21.15.70:36652) 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27118cd640 1 -- 172.21.15.70:0/2179029603 learned_addr learned my addr 172.21.15.70:0/2179029603 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27108cb640 1 -- 172.21.15.70:0/2179029603 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 653891681 0 0) 0x7f270c0b47a0 con 0x7f270c15e700 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27108cb640 1 -- 172.21.15.70:0/2179029603 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26f4003520 con 0x7f270c15e700 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27120ce640 1 -- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 msgr2=0x7f270c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27120ce640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 0x7f270c0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27120ce640 1 -- 172.21.15.70:0/2179029603 >> v1:172.21.15.96:6789/0 conn(0x7f270c15e700 legacy=0x7f270c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27120ce640 1 -- 172.21.15.70:0/2179029603 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c0b4d60 con 0x7f270c15eae0 2024-04-15T15:15:46.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.053+0000 7f27110cc640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 0x7f270c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.054+0000 7f27120ce640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0d30a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af677fedaa644443 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.054+0000 7f27108cb640 1 -- 172.21.15.70:0/2179029603 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc035020 con 0x7f270c15eae0 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.054+0000 7f27108cb640 1 -- 172.21.15.70:0/2179029603 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26fc031830 con 0x7f270c15eae0 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.054+0000 7f27108cb640 1 -- 172.21.15.70:0/2179029603 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26fc031c00 con 0x7f270c15eae0 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 -- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 msgr2=0x7f270c0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0d30a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f26fc0097a0 tx=0x7f26fc02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 -- 172.21.15.70:0/2179029603 shutdown_connections 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0d30a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 --2- 172.21.15.70:0/2179029603 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c0d2360 0x7f270c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 -- 172.21.15.70:0/2179029603 >> 172.21.15.70:0/2179029603 conn(0x7f270c08bf60 msgr2=0x7f270c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 -- 172.21.15.70:0/2179029603 shutdown_connections 2024-04-15T15:15:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.055+0000 7f2713b31640 1 -- 172.21.15.70:0/2179029603 wait complete. 2024-04-15T15:15:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.056+0000 7f2713b31640 1 Processor -- start 2024-04-15T15:15:46.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.056+0000 7f2713b31640 1 -- start start 2024-04-15T15:15:46.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c15e700 0x7f270c0b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f270c0d1d40 con 0x7f270c0d2360 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f270c0b4450 con 0x7f270c15eae0 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f2713b31640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f270c0d1ec0 con 0x7f270c15e700 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45258/0 (socket says 172.21.15.70:45258) 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 -- 172.21.15.70:0/4204972898 learned_addr learned my addr 172.21.15.70:0/4204972898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27110cc640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c15e700 0x7f270c0b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27120ce640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 -- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c15e700 msgr2=0x7f270c0b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c15e700 0x7f270c0b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 -- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 msgr2=0x7f270c0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 -- 172.21.15.70:0/4204972898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f270c0c8bd0 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.057+0000 7f27118cd640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f270400cbb0 tx=0x7f270400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.058+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704017070 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.058+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270400acc0 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.058+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704012640 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.058+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f270c0c8eb0 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.058+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f270c0c9470 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.059+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f270c0689e0 con 0x7f270c0d2360 2024-04-15T15:15:46.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.059+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f270401a070 con 0x7f270c0d2360 2024-04-15T15:15:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.059+0000 7f270a7fc640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 0x7f26f806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.060+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f270408bad0 con 0x7f270c0d2360 2024-04-15T15:15:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.060+0000 7f27110cc640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 0x7f26f806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.060+0000 7f27110cc640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 0x7f26f806edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2700009560 tx=0x7f2700009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.064+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2704059600 con 0x7f270c0d2360 2024-04-15T15:15:46.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.242+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f270c0d00f0 con 0x7f270c0d2360 2024-04-15T15:15:46.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.243+0000 7f270a7fc640 1 -- 172.21.15.70:0/4204972898 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f270405d900 con 0x7f270c0d2360 2024-04-15T15:15:46.245 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:46.245 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 msgr2=0x7f26f806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 0x7f26f806edb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2700009560 tx=0x7f2700009190 comp rx=0 tx=0).stop 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 msgr2=0x7f270c0c7f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f270400cbb0 tx=0x7f270400a060 comp rx=0 tx=0).stop 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 shutdown_connections 2024-04-15T15:15:46.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f270c15eae0 0x7f270c0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f270c15e700 0x7f270c0b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f26f806c8f0 0x7f26f806edb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 --2- 172.21.15.70:0/4204972898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f270c0d2360 0x7f270c0c7f50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 >> 172.21.15.70:0/4204972898 conn(0x7f270c08bf60 msgr2=0x7f270c0cf2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 shutdown_connections 2024-04-15T15:15:46.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.245+0000 7f2713b31640 1 -- 172.21.15.70:0/4204972898 wait complete. 2024-04-15T15:15:46.258 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:46.259 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 Processor -- start 2024-04-15T15:15:46.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 -- start start 2024-04-15T15:15:46.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6d9415ed00 0x7f6d94070920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6d940cf600 0x7f6d940cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d940ae440 con 0x7f6d9406ef70 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f6d940ae720 con 0x7f6d940cf600 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.367+0000 7f6d9918d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6d940aea00 con 0x7f6d9415ed00 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6d940cf600 0x7f6d940cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f6d940cf600 0x7f6d940cfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45272/0 (socket says 172.21.15.70:45272) 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 -- 172.21.15.70:0/3684854661 learned_addr learned my addr 172.21.15.70:0/3684854661 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d92d9d640 1 --2- 172.21.15.70:0/3684854661 >> v2:172.21.15.96:3300/0 conn(0x7f6d9415ed00 0x7f6d94070920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 -- 172.21.15.70:0/3684854661 >> v2:172.21.15.96:3300/0 conn(0x7f6d9415ed00 msgr2=0x7f6d94070920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3684854661 >> v2:172.21.15.96:3300/0 conn(0x7f6d9415ed00 0x7f6d94070920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 -- 172.21.15.70:0/3684854661 >> v1:172.21.15.96:6789/0 conn(0x7f6d9406ef70 legacy=0x7f6d9406f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.368+0000 7f6d9259c640 1 -- 172.21.15.70:0/3684854661 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d940aece0 con 0x7f6d940cf600 2024-04-15T15:15:46.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.369+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3684854661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d940cf600 0x7f6d940cfa90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6d940b4110 tx=0x7f6d8802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a897e93678c926 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.369+0000 7f6d91d9b640 1 -- 172.21.15.70:0/3684854661 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88034070 con 0x7f6d940cf600 2024-04-15T15:15:46.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.369+0000 7f6d91d9b640 1 -- 172.21.15.70:0/3684854661 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d88030860 con 0x7f6d940cf600 2024-04-15T15:15:46.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.369+0000 7f6d91d9b640 1 -- 172.21.15.70:0/3684854661 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d88030c30 con 0x7f6d940cf600 2024-04-15T15:15:46.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 -- 172.21.15.70:0/3684854661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d940cf600 msgr2=0x7f6d940cfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3684854661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d940cf600 0x7f6d940cfa90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6d940b4110 tx=0x7f6d8802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 -- 172.21.15.70:0/3684854661 shutdown_connections 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3684854661 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d940cf600 0x7f6d940cfa90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3684854661 >> v2:172.21.15.96:3300/0 conn(0x7f6d9415ed00 0x7f6d94070920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 -- 172.21.15.70:0/3684854661 >> 172.21.15.70:0/3684854661 conn(0x7f6d9409a9d0 msgr2=0x7f6d940d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 -- 172.21.15.70:0/3684854661 shutdown_connections 2024-04-15T15:15:46.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.370+0000 7f6d9918d640 1 -- 172.21.15.70:0/3684854661 wait complete. 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.371+0000 7f6d9918d640 1 Processor -- start 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.371+0000 7f6d9918d640 1 -- start start 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 0x7f6d94154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 0x7f6d940b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6d94070e60 con 0x7f6d9406ef70 2024-04-15T15:15:46.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6d940701a0 con 0x7f6d940cf600 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d9918d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6d94070fe0 con 0x7f6d9415ed00 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d92d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d92d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45280/0 (socket says 172.21.15.70:45280) 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.372+0000 7f6d92d9d640 1 -- 172.21.15.70:0/3888495291 learned_addr learned my addr 172.21.15.70:0/3888495291 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 0x7f6d94154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9359e640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 0x7f6d940b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 -- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 msgr2=0x7f6d940b30b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 0x7f6d940b30b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 -- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 msgr2=0x7f6d94154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 0x7f6d94154210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 -- 172.21.15.70:0/3888495291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d940b54a0 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9359e640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 0x7f6d940b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 0x7f6d94154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d92d9d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400e930 tx=0x7f6d8400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d84017070 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d8400ca00 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d940b5780 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d8400cdf0 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.373+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d940b5d40 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.374+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d940690f0 con 0x7f6d9406ef70 2024-04-15T15:15:46.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.375+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6d8401a070 con 0x7f6d9406ef70 2024-04-15T15:15:46.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.375+0000 7f6d7b7fe640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 0x7f6d8006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.375+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6d8408c220 con 0x7f6d9406ef70 2024-04-15T15:15:46.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.375+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 0x7f6d8006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.376+0000 7f6d9259c640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 0x7f6d8006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d88036000 tx=0x7f6d8803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.380+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6d8405a460 con 0x7f6d9406ef70 2024-04-15T15:15:46.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.566+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6d9415e250 con 0x7f6d9406ef70 2024-04-15T15:15:46.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.566+0000 7f6d7b7fe640 1 -- 172.21.15.70:0/3888495291 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f6d84059d60 con 0x7f6d9406ef70 2024-04-15T15:15:46.569 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:46.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 msgr2=0x7f6d8006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 0x7f6d8006edb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6d88036000 tx=0x7f6d8803b040 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 msgr2=0x7f6d940c50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f6d8400e930 tx=0x7f6d8400edf0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 shutdown_connections 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6d9415ed00 0x7f6d940b30b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6d940cf600 0x7f6d94154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6d8006c8f0 0x7f6d8006edb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.568+0000 7f6d9918d640 1 --2- 172.21.15.70:0/3888495291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6d9406ef70 0x7f6d940c50c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.569+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 >> 172.21.15.70:0/3888495291 conn(0x7f6d9409a9d0 msgr2=0x7f6d940cbdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.569+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 shutdown_connections 2024-04-15T15:15:46.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.569+0000 7f6d9918d640 1 -- 172.21.15.70:0/3888495291 wait complete. 2024-04-15T15:15:46.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:46.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.690+0000 7f5dd84eb640 1 Processor -- start 2024-04-15T15:15:46.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.690+0000 7f5dd84eb640 1 -- start start 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5dd015e790 0x7f5dd00b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5dd00b7be0 0x7f5dd00d15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 0x7f5dd00d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd00d1f90 con 0x7f5dd00b7be0 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5dd00d3790 con 0x7f5dd015e790 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd84eb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd00d45b0 con 0x7f5dd00d1b20 2024-04-15T15:15:46.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd5a86640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5dd00b7be0 0x7f5dd00d15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd5a86640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5dd00b7be0 0x7f5dd00d15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45296/0 (socket says 172.21.15.70:45296) 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.691+0000 7f5dd5a86640 1 -- 172.21.15.70:0/2939992034 learned_addr learned my addr 172.21.15.70:0/2939992034 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd6287640 1 --2- 172.21.15.70:0/2939992034 >> v2:172.21.15.96:3300/0 conn(0x7f5dd015e790 0x7f5dd00b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd6a88640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 0x7f5dd00d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd5a86640 1 -- 172.21.15.70:0/2939992034 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 msgr2=0x7f5dd00d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 0x7f5dd00d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd5a86640 1 -- 172.21.15.70:0/2939992034 >> v2:172.21.15.96:3300/0 conn(0x7f5dd015e790 msgr2=0x7f5dd00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/2939992034 >> v2:172.21.15.96:3300/0 conn(0x7f5dd015e790 0x7f5dd00b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd5a86640 1 -- 172.21.15.70:0/2939992034 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd015eb70 con 0x7f5dd00b7be0 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.692+0000 7f5dd6a88640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 0x7f5dd00d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00d15e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00d4730 tx=0x7f5dc002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce95205a06ad4fc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd5285640 1 -- 172.21.15.70:0/2939992034 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0038070 con 0x7f5dd00b7be0 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd5285640 1 -- 172.21.15.70:0/2939992034 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc0031740 con 0x7f5dd00b7be0 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd5285640 1 -- 172.21.15.70:0/2939992034 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc002f050 con 0x7f5dd00b7be0 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 -- 172.21.15.70:0/2939992034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00b7be0 msgr2=0x7f5dd00d15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00d15e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5dd00d4730 tx=0x7f5dc002b7e0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 -- 172.21.15.70:0/2939992034 shutdown_connections 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd00d1b20 0x7f5dd00d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/2939992034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00d15e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/2939992034 >> v2:172.21.15.96:3300/0 conn(0x7f5dd015e790 0x7f5dd00b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.693+0000 7f5dd84eb640 1 -- 172.21.15.70:0/2939992034 >> 172.21.15.70:0/2939992034 conn(0x7f5dd0067b40 msgr2=0x7f5dd0067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.694+0000 7f5dd84eb640 1 -- 172.21.15.70:0/2939992034 shutdown_connections 2024-04-15T15:15:46.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.694+0000 7f5dd84eb640 1 -- 172.21.15.70:0/2939992034 wait complete. 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 Processor -- start 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 -- start start 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00cded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 0x7f5dd00ce920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd00d4240 con 0x7f5dd00d1b20 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5dd00d3f60 con 0x7f5dd00b7be0 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd84eb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5dd00d4f80 con 0x7f5dd015e790 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.695+0000 7f5dd5a86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45312/0 (socket says 172.21.15.70:45312) 2024-04-15T15:15:46.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 -- 172.21.15.70:0/1171125618 learned_addr learned my addr 172.21.15.70:0/1171125618 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd6287640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd6a88640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 0x7f5dd00ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 -- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 msgr2=0x7f5dd00ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 0x7f5dd00ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 -- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dd00b7be0 msgr2=0x7f5dd00cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00cded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 -- 172.21.15.70:0/1171125618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dd00c8490 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd6a88640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 0x7f5dd00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd5a86640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5dc002b820 tx=0x7f5dc00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0042070 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dc0031e80 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dd00c8770 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0038040 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.696+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dd00c8d30 con 0x7f5dd00d1b20 2024-04-15T15:15:46.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.697+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dd00b6820 con 0x7f5dd00d1b20 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.698+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f5dc0045070 con 0x7f5dd00d1b20 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.698+0000 7f5dc6ffd640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 0x7f5db406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.698+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f5dc00adb30 con 0x7f5dd00d1b20 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.698+0000 7f5dd6287640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 0x7f5db406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.698+0000 7f5dd6287640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 0x7f5db406edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc006fd0 tx=0x7f5dcc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:46.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.702+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5dd00b6820 con 0x7f5dd00d1b20 2024-04-15T15:15:46.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.879+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5dd00689e0 con 0x7f5dd00d1b20 2024-04-15T15:15:46.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.879+0000 7f5dc6ffd640 1 -- 172.21.15.70:0/1171125618 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f5dc004c030 con 0x7f5dd00d1b20 2024-04-15T15:15:46.881 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:46.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 msgr2=0x7f5db406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 0x7f5db406edb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5dcc006fd0 tx=0x7f5dcc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 msgr2=0x7f5dd00c9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5dc002b820 tx=0x7f5dc00046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 shutdown_connections 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5dd015e790 0x7f5dd00ce920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5db406c8f0 0x7f5db406edb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5dd00d1b20 0x7f5dd00c9e70 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 --2- 172.21.15.70:0/1171125618 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5dd00b7be0 0x7f5dd00cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.881+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 >> 172.21.15.70:0/1171125618 conn(0x7f5dd0067b40 msgr2=0x7f5dd00b98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.882+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 shutdown_connections 2024-04-15T15:15:46.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:46.882+0000 7f5dd84eb640 1 -- 172.21.15.70:0/1171125618 wait complete. 2024-04-15T15:15:46.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:47.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.000+0000 7fa45660d640 1 Processor -- start 2024-04-15T15:15:47.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 -- start start 2024-04-15T15:15:47.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4500d2360 0x7fa4500d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4500b47a0 con 0x7fa45015e700 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa4500b53b0 con 0x7fa4500d2360 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa45660d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500b4a80 con 0x7fa45015eae0 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.001+0000 7fa44f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4500d2360 0x7fa4500d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa4500d2360 0x7fa4500d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45314/0 (socket says 172.21.15.70:45314) 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 -- 172.21.15.70:0/2030788058 learned_addr learned my addr 172.21.15.70:0/2030788058 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa454baa640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 -- 172.21.15.70:0/2030788058 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 msgr2=0x7fa4500d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 -- 172.21.15.70:0/2030788058 >> v1:172.21.15.70:6789/0 conn(0x7fa45015e700 legacy=0x7fa4500b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44effd640 1 -- 172.21.15.70:0/2030788058 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631178077 0 0) 0x7fa4500b47a0 con 0x7fa45015e700 2024-04-15T15:15:47.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.002+0000 7fa44f7fe640 1 -- 172.21.15.70:0/2030788058 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4500b4d60 con 0x7fa4500d2360 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500d2b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa444009870 tx=0x7fa44402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb184938c2a8c7e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa44effd640 1 -- 172.21.15.70:0/2030788058 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa444034070 con 0x7fa4500d2360 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa44effd640 1 -- 172.21.15.70:0/2030788058 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa444030840 con 0x7fa4500d2360 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa44effd640 1 -- 172.21.15.70:0/2030788058 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa444030c10 con 0x7fa4500d2360 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 -- 172.21.15.70:0/2030788058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 msgr2=0x7fa4500d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500d2b60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa444009870 tx=0x7fa44402b980 comp rx=0 tx=0).stop 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 -- 172.21.15.70:0/2030788058 shutdown_connections 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 --2- 172.21.15.70:0/2030788058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500d2b60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 -- 172.21.15.70:0/2030788058 >> 172.21.15.70:0/2030788058 conn(0x7fa45008bf60 msgr2=0x7fa4500d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 -- 172.21.15.70:0/2030788058 shutdown_connections 2024-04-15T15:15:47.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.003+0000 7fa45660d640 1 -- 172.21.15.70:0/2030788058 wait complete. 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.004+0000 7fa45660d640 1 Processor -- start 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 -- start start 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 0x7fa4501618b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500d1d40 con 0x7fa4500d2360 2024-04-15T15:15:47.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa4500b4450 con 0x7fa45015e700 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa45660d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa4500d1ec0 con 0x7fa45015eae0 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.005+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45318/0 (socket says 172.21.15.70:45318) 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 -- 172.21.15.70:0/732901578 learned_addr learned my addr 172.21.15.70:0/732901578 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa454baa640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 0x7fa4501618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 -- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 msgr2=0x7fa4500b8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500b8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 -- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 msgr2=0x7fa4501618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 0x7fa4501618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 -- 172.21.15.70:0/732901578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4500b8080 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa454baa640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 0x7fa4501618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44ffff640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c400 tx=0x7fa44000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440017070 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440003610 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4500b8320 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440003a20 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.006+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4500cd1e0 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.007+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4500689e0 con 0x7fa4500d2360 2024-04-15T15:15:47.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.008+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa44001a070 con 0x7fa4500d2360 2024-04-15T15:15:47.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.008+0000 7fa44cff9640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 0x7fa43c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.008+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa44008b220 con 0x7fa4500d2360 2024-04-15T15:15:47.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.008+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 0x7fa43c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.009+0000 7fa44f7fe640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 0x7fa43c06edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa444036000 tx=0x7fa44403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.017+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa4400593a0 con 0x7fa4500d2360 2024-04-15T15:15:47.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.199+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa4500caee0 con 0x7fa4500d2360 2024-04-15T15:15:47.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.199+0000 7fa44cff9640 1 -- 172.21.15.70:0/732901578 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fa440058ca0 con 0x7fa4500d2360 2024-04-15T15:15:47.201 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 msgr2=0x7fa43c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 0x7fa43c06edb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa444036000 tx=0x7fa44403b040 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 msgr2=0x7fa4500cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa44000c400 tx=0x7fa44000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 shutdown_connections 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa45015eae0 0x7fa4500b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa45015e700 0x7fa4501618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa43c06c8f0 0x7fa43c06edb0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 --2- 172.21.15.70:0/732901578 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4500d2360 0x7fa4500cdca0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.201+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 >> 172.21.15.70:0/732901578 conn(0x7fa45008bf60 msgr2=0x7fa4500b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.202+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 shutdown_connections 2024-04-15T15:15:47.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.202+0000 7fa45660d640 1 -- 172.21.15.70:0/732901578 wait complete. 2024-04-15T15:15:47.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:47.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.319+0000 7f4fa024a640 1 Processor -- start 2024-04-15T15:15:47.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.320+0000 7f4fa024a640 1 -- start start 2024-04-15T15:15:47.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.320+0000 7f4fa024a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f9806c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.320+0000 7f4fa024a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4f9806ecb0 0x7f4f980b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4fa024a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f9806f220 con 0x7f4f9806cf30 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4fa024a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4f9806d390 con 0x7f4f9806ecb0 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4fa024a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9806f500 con 0x7f4f9815e6f0 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9d7e5640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4f9806cf30 0x7f4f9806e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55862/0 (socket says 172.21.15.70:55862) 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9d7e5640 1 -- 172.21.15.70:0/902932354 learned_addr learned my addr 172.21.15.70:0/902932354 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:47.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/902932354 >> v2:172.21.15.96:3300/0 conn(0x7f4f9806ecb0 0x7f4f980b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f9806c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 171150347 0 0) 0x7f4f9806f220 con 0x7f4f9806cf30 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f84003520 con 0x7f4f9806cf30 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4123061767 0 0) 0x7f4f84003520 con 0x7f4f9806cf30 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 msgr2=0x7f4f9806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f9806c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 >> v2:172.21.15.96:3300/0 conn(0x7f4f9806ecb0 msgr2=0x7f4f980b4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 --2- 172.21.15.70:0/902932354 >> v2:172.21.15.96:3300/0 conn(0x7f4f9806ecb0 0x7f4f980b4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f980b45d0 con 0x7f4f9806cf30 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4f88001670 con 0x7f4f9806cf30 2024-04-15T15:15:47.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 >> v1:172.21.15.70:6789/0 conn(0x7f4f9806cf30 legacy=0x7f4f9806e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84003f30 0x7f4f84004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84004d40 0x7f4f840051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f9806f500 con 0x7f4f84003f30 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f8400b9c0 con 0x7f4f84004d40 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f8400bb40 con 0x7f4f9815e6f0 2024-04-15T15:15:47.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.321+0000 7f4f9d7e5640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84003f30 0x7f4f84004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84004d40 0x7f4f840051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 -- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 msgr2=0x7f4f840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f9815e6f0 0x7f4f840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 -- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84004d40 msgr2=0x7f4f840051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84004d40 0x7f4f840051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 -- 172.21.15.70:0/902932354 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f84003520 con 0x7f4f84003f30 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f84004d40 0x7f4f840051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.322+0000 7f4f9d7e5640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84003f30 0x7f4f84004800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4f88007fd0 tx=0x7f4f84005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80a8a2a37bfabba1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f88007d50 con 0x7f4f84003f30 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f88030070 con 0x7f4f84003f30 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4f9cfe4640 1 -- 172.21.15.70:0/902932354 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8802cd20 con 0x7f4f84003f30 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 -- 172.21.15.70:0/902932354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84003f30 msgr2=0x7f4f84004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 --2- 172.21.15.70:0/902932354 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f84003f30 0x7f4f84004800 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4f88007fd0 tx=0x7f4f84005830 comp rx=0 tx=0).stop 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4f9e7e7640 1 -- 172.21.15.70:0/902932354 reap_dead start 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 -- 172.21.15.70:0/902932354 shutdown_connections 2024-04-15T15:15:47.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 -- 172.21.15.70:0/902932354 >> 172.21.15.70:0/902932354 conn(0x7f4f98072440 msgr2=0x7f4f98072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 -- 172.21.15.70:0/902932354 shutdown_connections 2024-04-15T15:15:47.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.323+0000 7f4fa024a640 1 -- 172.21.15.70:0/902932354 wait complete. 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.325+0000 7f4fa024a640 1 Processor -- start 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.325+0000 7f4fa024a640 1 -- start start 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 0x7f4f98161a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9814a6d0 0x7f4f981f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980b4370 con 0x7f4f98157d30 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4f980b41f0 con 0x7f4f9814a6d0 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4fa024a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9815ead0 con 0x7f4f84003f30 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.326+0000 7f4f9e7e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45336/0 (socket says 172.21.15.70:45336) 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 -- 172.21.15.70:0/3184663381 learned_addr learned my addr 172.21.15.70:0/3184663381 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9d7e5640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9814a6d0 0x7f4f981f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 0x7f4f98161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 -- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 msgr2=0x7f4f98161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 0x7f4f98161a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 -- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9814a6d0 msgr2=0x7f4f981f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9814a6d0 0x7f4f981f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 -- 172.21.15.70:0/3184663381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f981f4f40 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 0x7f4f98161a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f9e7e7640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f9806f430 tx=0x7f4f90010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90010d80 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f90003990 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f981f5220 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90003d80 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.327+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f98157810 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.328+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f981605b0 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.329+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4f90005d10 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.329+0000 7f4f8e7fc640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 0x7f4f7c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.329+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4f9008c690 con 0x7f4f98157d30 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.329+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 0x7f4f7c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.330+0000 7f4f9dfe6640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 0x7f4f7c06ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009ae0 tx=0x7f4f94008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.338+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4f981605b0 con 0x7f4f98157d30 2024-04-15T15:15:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.520+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4f98157600 con 0x7f4f98157d30 2024-04-15T15:15:47.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.521+0000 7f4f8e7fc640 1 -- 172.21.15.70:0/3184663381 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f4f90025020 con 0x7f4f98157d30 2024-04-15T15:15:47.523 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 msgr2=0x7f4f7c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 0x7f4f7c06ede0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4f94009ae0 tx=0x7f4f94008040 comp rx=0 tx=0).stop 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 msgr2=0x7f4f981581b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4f9806f430 tx=0x7f4f90010590 comp rx=0 tx=0).stop 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 shutdown_connections 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4f7c06c920 0x7f4f7c06ede0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4f98157d30 0x7f4f981581b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4f9814a6d0 0x7f4f981f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 --2- 172.21.15.70:0/3184663381 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4f84003f30 0x7f4f98161a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 >> 172.21.15.70:0/3184663381 conn(0x7f4f98072440 msgr2=0x7f4f980cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 shutdown_connections 2024-04-15T15:15:47.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.523+0000 7f4fa024a640 1 -- 172.21.15.70:0/3184663381 wait complete. 2024-04-15T15:15:47.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.637+0000 7f8adc6fb640 1 Processor -- start 2024-04-15T15:15:47.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.637+0000 7f8adc6fb640 1 -- start start 2024-04-15T15:15:47.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adc6fb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8ad40d3270 0x7f8ad40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adc6fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40b47a0 con 0x7f8ad40d2360 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adc6fb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ad40b4a80 con 0x7f8ad415e700 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adc6fb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8ad40b53b0 con 0x7f8ad40d3270 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8ad40d3270 0x7f8ad40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8ad40d3270 0x7f8ad40cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45352/0 (socket says 172.21.15.70:45352) 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8ad9c96640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8ad40d2360 0x7f8ad40d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55866/0 (socket says 172.21.15.70:55866) 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 -- 172.21.15.70:0/2980360745 learned_addr learned my addr 172.21.15.70:0/2980360745 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 -- 172.21.15.70:0/2980360745 >> v1:172.21.15.96:6789/0 conn(0x7f8ad415e700 legacy=0x7f8ad40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 -- 172.21.15.70:0/2980360745 >> v1:172.21.15.70:6789/0 conn(0x7f8ad40d2360 legacy=0x7f8ad40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8adac98640 1 -- 172.21.15.70:0/2980360745 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40b4d60 con 0x7f8ad40d3270 2024-04-15T15:15:47.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.638+0000 7f8ad9495640 1 -- 172.21.15.70:0/2980360745 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 713617196 0 0) 0x7f8ad40b47a0 con 0x7f8ad40d2360 2024-04-15T15:15:47.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8adac98640 1 --2- 172.21.15.70:0/2980360745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40cf370 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40b5530 tx=0x7f8acc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649ee6d7e610b03f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8ad9495640 1 -- 172.21.15.70:0/2980360745 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc034070 con 0x7f8ad40d3270 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8ad9495640 1 -- 172.21.15.70:0/2980360745 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8acc030840 con 0x7f8ad40d3270 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8ad9495640 1 -- 172.21.15.70:0/2980360745 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8acc030c10 con 0x7f8ad40d3270 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2980360745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 msgr2=0x7f8ad40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.639+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2980360745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40cf370 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40b5530 tx=0x7f8acc02b980 comp rx=0 tx=0).stop 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.640+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2980360745 shutdown_connections 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.640+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2980360745 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40cf370 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.640+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2980360745 >> 172.21.15.70:0/2980360745 conn(0x7f8ad408bf60 msgr2=0x7f8ad40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.640+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2980360745 shutdown_connections 2024-04-15T15:15:47.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.640+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2980360745 wait complete. 2024-04-15T15:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 Processor -- start 2024-04-15T15:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 -- start start 2024-04-15T15:15:47.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 0x7f8ad40b8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ad415e700 0x7f8ad415f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40d1d40 con 0x7f8ad40d3270 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8ad40b4450 con 0x7f8ad40d2360 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.641+0000 7f8adc6fb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8ad40d1ec0 con 0x7f8ad415e700 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45364/0 (socket says 172.21.15.70:45364) 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 -- 172.21.15.70:0/2261059808 learned_addr learned my addr 172.21.15.70:0/2261059808 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ada497640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 0x7f8ad40b8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8adac98640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ad415e700 0x7f8ad415f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 -- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ad415e700 msgr2=0x7f8ad415f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ad415e700 0x7f8ad415f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 -- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 msgr2=0x7f8ad40b8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 0x7f8ad40b8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 -- 172.21.15.70:0/2261059808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ad40b9b40 con 0x7f8ad40d3270 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ada497640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 0x7f8ad40b8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8ad9c96640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c4d0 tx=0x7f8ac400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4018070 con 0x7f8ad40d3270 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ac4007500 con 0x7f8ad40d3270 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ad40b92a0 con 0x7f8ad40d3270 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4004a20 con 0x7f8ad40d3270 2024-04-15T15:15:47.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.642+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ad40cc1c0 con 0x7f8ad40d3270 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.643+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ad40b9690 con 0x7f8ad40d3270 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.644+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8ac401b070 con 0x7f8ad40d3270 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.644+0000 7f8acaffd640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 0x7f8ab806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.644+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8ac408b190 con 0x7f8ad40d3270 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.644+0000 7f8ada497640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 0x7f8ab806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.644+0000 7f8ada497640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 0x7f8ab806edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000c420 tx=0x7f8ad0009570 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.648+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8ac4059280 con 0x7f8ad40d3270 2024-04-15T15:15:47.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.824+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8ad40cfb10 con 0x7f8ad40d3270 2024-04-15T15:15:47.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.826+0000 7f8acaffd640 1 -- 172.21.15.70:0/2261059808 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f8ad40cfb10 con 0x7f8ad40d3270 2024-04-15T15:15:47.827 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:47.828 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:47.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.827+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 msgr2=0x7f8ab806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.827+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 0x7f8ab806edb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000c420 tx=0x7f8ad0009570 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.827+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 msgr2=0x7f8ad40a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.827+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c4d0 tx=0x7f8ac400c990 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 shutdown_connections 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8ad415e700 0x7f8ad415f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8ab806c8f0 0x7f8ab806edb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ad40d3270 0x7f8ad40a1dc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 --2- 172.21.15.70:0/2261059808 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8ad40d2360 0x7f8ad40b8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 >> 172.21.15.70:0/2261059808 conn(0x7f8ad408bf60 msgr2=0x7f8ad40b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 shutdown_connections 2024-04-15T15:15:47.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.828+0000 7f8adc6fb640 1 -- 172.21.15.70:0/2261059808 wait complete. 2024-04-15T15:15:47.840 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:47.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.944+0000 7f34154c1640 1 Processor -- start 2024-04-15T15:15:47.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.944+0000 7f34154c1640 1 -- start start 2024-04-15T15:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34154c1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 0x7f3410071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34154c1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f34100af3e0 0x7f34100af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34154c1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341015db40 con 0x7f34100d0690 2024-04-15T15:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34154c1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f341015cfc0 con 0x7f34100af3e0 2024-04-15T15:15:47.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34154c1640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f341015d2a0 con 0x7f3410072d80 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34067fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f34100af3e0 0x7f34100af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34067fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f34100af3e0 0x7f34100af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45370/0 (socket says 172.21.15.70:45370) 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f340f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f34100d0690 0x7f341006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55876/0 (socket says 172.21.15.70:55876) 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.945+0000 7f34067fc640 1 -- 172.21.15.70:0/1419235522 learned_addr learned my addr 172.21.15.70:0/1419235522 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f340effd640 1 --2- 172.21.15.70:0/1419235522 >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 0x7f3410071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f34067fc640 1 -- 172.21.15.70:0/1419235522 >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 msgr2=0x7f3410071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f34067fc640 1 --2- 172.21.15.70:0/1419235522 >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 0x7f3410071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f34067fc640 1 -- 172.21.15.70:0/1419235522 >> v1:172.21.15.70:6789/0 conn(0x7f34100d0690 legacy=0x7f341006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f34067fc640 1 -- 172.21.15.70:0/1419235522 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341015d580 con 0x7f34100af3e0 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f340e7fc640 1 -- 172.21.15.70:0/1419235522 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750898844 0 0) 0x7f341015db40 con 0x7f34100d0690 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f340effd640 1 --2- 172.21.15.70:0/1419235522 >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 0x7f3410071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.946+0000 7f34067fc640 1 --2- 172.21.15.70:0/1419235522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100af3e0 0x7f34100af840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f34100d11c0 tx=0x7f33f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5749507fbf9e9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f340e7fc640 1 -- 172.21.15.70:0/1419235522 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8035020 con 0x7f34100af3e0 2024-04-15T15:15:47.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f340e7fc640 1 -- 172.21.15.70:0/1419235522 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33f8031870 con 0x7f34100af3e0 2024-04-15T15:15:47.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f340e7fc640 1 -- 172.21.15.70:0/1419235522 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33f8031c40 con 0x7f34100af3e0 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 -- 172.21.15.70:0/1419235522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100af3e0 msgr2=0x7f34100af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 --2- 172.21.15.70:0/1419235522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100af3e0 0x7f34100af840 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f34100d11c0 tx=0x7f33f802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 -- 172.21.15.70:0/1419235522 shutdown_connections 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 --2- 172.21.15.70:0/1419235522 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100af3e0 0x7f34100af840 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 --2- 172.21.15.70:0/1419235522 >> v2:172.21.15.96:3300/0 conn(0x7f3410072d80 0x7f3410071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 -- 172.21.15.70:0/1419235522 >> 172.21.15.70:0/1419235522 conn(0x7f341006be00 msgr2=0x7f341006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 -- 172.21.15.70:0/1419235522 shutdown_connections 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.947+0000 7f34154c1640 1 -- 172.21.15.70:0/1419235522 wait complete. 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.948+0000 7f34154c1640 1 Processor -- start 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 -- start start 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3410072d80 0x7f34100b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34100af3e0 0x7f34101552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f341015cd10 con 0x7f34100d0690 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f341015cb90 con 0x7f34100af3e0 2024-04-15T15:15:47.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f34154c1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f34100d0b00 con 0x7f3410072d80 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f340f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45382/0 (socket says 172.21.15.70:45382) 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.949+0000 7f340f7fe640 1 -- 172.21.15.70:0/2636301273 learned_addr learned my addr 172.21.15.70:0/2636301273 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340effd640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3410072d80 0x7f34100b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 -- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3410072d80 msgr2=0x7f34100b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f34067fc640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34100af3e0 0x7f34101552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3410072d80 0x7f34100b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 -- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34100af3e0 msgr2=0x7f34101552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34100af3e0 0x7f34101552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:47.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 -- 172.21.15.70:0/2636301273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34100c9680 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f340f7fe640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f340800c700 tx=0x7f340800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3408018070 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f340800ce10 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340800a800 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34100c9960 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.950+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34100ca440 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.951+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34100689e0 con 0x7f34100d0690 2024-04-15T15:15:47.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.951+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f340800a9a0 con 0x7f34100d0690 2024-04-15T15:15:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.952+0000 7f3407fff640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 0x7f33f006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.952+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f340808b490 con 0x7f34100d0690 2024-04-15T15:15:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.952+0000 7f340effd640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 0x7f33f006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:47.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.952+0000 7f340effd640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 0x7f33f006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3400008850 tx=0x7f3400008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:47.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:47.956+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3408059580 con 0x7f34100d0690 2024-04-15T15:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.132+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f341015f800 con 0x7f34100d0690 2024-04-15T15:15:48.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.132+0000 7f3407fff640 1 -- 172.21.15.70:0/2636301273 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f3408058e80 con 0x7f34100d0690 2024-04-15T15:15:48.134 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:48.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 msgr2=0x7f33f006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 0x7f33f006edb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3400008850 tx=0x7f3400008040 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 msgr2=0x7f34101557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f340800c700 tx=0x7f340800cbc0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 shutdown_connections 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f33f006c8f0 0x7f33f006edb0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f34100d0690 0x7f34101557e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f34100af3e0 0x7f34101552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 --2- 172.21.15.70:0/2636301273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3410072d80 0x7f34100b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 >> 172.21.15.70:0/2636301273 conn(0x7f341006be00 msgr2=0x7f3410074c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.134+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 shutdown_connections 2024-04-15T15:15:48.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.135+0000 7f34154c1640 1 -- 172.21.15.70:0/2636301273 wait complete. 2024-04-15T15:15:48.147 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:48.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.251+0000 7fa1f0596640 1 Processor -- start 2024-04-15T15:15:48.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.251+0000 7fa1f0596640 1 -- start start 2024-04-15T15:15:48.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1f0596640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 0x7fa1e80b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1f0596640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa1e80d1a10 0x7fa1e80b1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1f0596640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1e815ba70 con 0x7fa1e80d08a0 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1f0596640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa1e80ce5b0 con 0x7fa1e80d1a10 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1f0596640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa1e80ce890 con 0x7fa1e815b690 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1eeb33640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa1e80d1a10 0x7fa1e80b1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1eeb33640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa1e80d1a10 0x7fa1e80b1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45396/0 (socket says 172.21.15.70:45396) 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.252+0000 7fa1eeb33640 1 -- 172.21.15.70:0/2426276963 learned_addr learned my addr 172.21.15.70:0/2426276963 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1eeb33640 1 -- 172.21.15.70:0/2426276963 >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 msgr2=0x7fa1e80b6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1ee332640 1 --2- 172.21.15.70:0/2426276963 >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 0x7fa1e80b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1eeb33640 1 --2- 172.21.15.70:0/2426276963 >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 0x7fa1e80b6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1eeb33640 1 -- 172.21.15.70:0/2426276963 >> v1:172.21.15.96:6789/0 conn(0x7fa1e80d08a0 legacy=0x7fa1e80d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1eeb33640 1 -- 172.21.15.70:0/2426276963 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e80ceb70 con 0x7fa1e80d1a10 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.253+0000 7fa1ee332640 1 --2- 172.21.15.70:0/2426276963 >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 0x7fa1e80b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1eeb33640 1 --2- 172.21.15.70:0/2426276963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d1a10 0x7fa1e80b1660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009870 tx=0x7fa1e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeae32dba264cce9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1ed330640 1 -- 172.21.15.70:0/2426276963 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0035020 con 0x7fa1e80d1a10 2024-04-15T15:15:48.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1ed330640 1 -- 172.21.15.70:0/2426276963 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1e0031870 con 0x7fa1e80d1a10 2024-04-15T15:15:48.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1ed330640 1 -- 172.21.15.70:0/2426276963 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0031c40 con 0x7fa1e80d1a10 2024-04-15T15:15:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 -- 172.21.15.70:0/2426276963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d1a10 msgr2=0x7fa1e80b1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 --2- 172.21.15.70:0/2426276963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d1a10 0x7fa1e80b1660 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009870 tx=0x7fa1e002b910 comp rx=0 tx=0).stop 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 -- 172.21.15.70:0/2426276963 shutdown_connections 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 --2- 172.21.15.70:0/2426276963 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d1a10 0x7fa1e80b1660 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 --2- 172.21.15.70:0/2426276963 >> v2:172.21.15.96:3300/0 conn(0x7fa1e815b690 0x7fa1e80b6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 -- 172.21.15.70:0/2426276963 >> 172.21.15.70:0/2426276963 conn(0x7fa1e809a9d0 msgr2=0x7fa1e80b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 -- 172.21.15.70:0/2426276963 shutdown_connections 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.254+0000 7fa1f0596640 1 -- 172.21.15.70:0/2426276963 wait complete. 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 Processor -- start 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 -- start start 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 0x7fa1e809c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 0x7fa1e80b56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e80b1ba0 con 0x7fa1e80d08a0 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e80d1010 con 0x7fa1e815b690 2024-04-15T15:15:48.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1f0596640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa1e80b1d20 con 0x7fa1e80d1a10 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1ee332640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1ee332640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45408/0 (socket says 172.21.15.70:45408) 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.256+0000 7fa1ee332640 1 -- 172.21.15.70:0/1778910561 learned_addr learned my addr 172.21.15.70:0/1778910561 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1eeb33640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 0x7fa1e80b56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1edb31640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 0x7fa1e809c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 -- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 msgr2=0x7fa1e809c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 0x7fa1e809c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 -- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 msgr2=0x7fa1e80b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 0x7fa1e80b56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 -- 172.21.15.70:0/1778910561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e809b550 con 0x7fa1e80d08a0 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1edb31640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 0x7fa1e809c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1eeb33640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 0x7fa1e80b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1ee332640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800ce00 tx=0x7fa1d800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800f680 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d800f820 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800fc30 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e809b830 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.257+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e809bdf0 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.258+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1e80690f0 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.258+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa1d8018400 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.259+0000 7fa1e6ffd640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 0x7fa1d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.259+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa1d808cc20 con 0x7fa1e80d08a0 2024-04-15T15:15:48.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.259+0000 7fa1edb31640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 0x7fa1d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.259+0000 7fa1edb31640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 0x7fa1d406edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80b5660 tx=0x7fa1dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.263+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa1d8014030 con 0x7fa1e80d08a0 2024-04-15T15:15:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.442+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa1e809c1f0 con 0x7fa1e80d08a0 2024-04-15T15:15:48.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.443+0000 7fa1e6ffd640 1 -- 172.21.15.70:0/1778910561 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fa1d805a750 con 0x7fa1e80d08a0 2024-04-15T15:15:48.445 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 msgr2=0x7fa1d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 0x7fa1d406edb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa1e80b5660 tx=0x7fa1dc009040 comp rx=0 tx=0).stop 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 msgr2=0x7fa1e806bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800ce00 tx=0x7fa1d800a650 comp rx=0 tx=0).stop 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 shutdown_connections 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa1e815b690 0x7fa1e80b56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa1e80d1a10 0x7fa1e809c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1d406c8f0 0x7fa1d406edb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 --2- 172.21.15.70:0/1778910561 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa1e80d08a0 0x7fa1e806bc50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 >> 172.21.15.70:0/1778910561 conn(0x7fa1e809a9d0 msgr2=0x7fa1e815a160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 shutdown_connections 2024-04-15T15:15:48.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.445+0000 7fa1f0596640 1 -- 172.21.15.70:0/1778910561 wait complete. 2024-04-15T15:15:48.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:48.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.561+0000 7efd91de2640 1 Processor -- start 2024-04-15T15:15:48.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.561+0000 7efd91de2640 1 -- start start 2024-04-15T15:15:48.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd91de2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd91de2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 0x7efd8c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd91de2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd8c070720 con 0x7efd8c09adb0 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd91de2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efd8c09ba90 con 0x7efd8c0701f0 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd91de2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c160070 con 0x7efd8c09a9d0 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd8b7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd8affd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 0x7efd8c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd8bfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7efd8c09adb0 0x7efd8c0d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:36684/0 (socket says 172.21.15.70:36684) 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd8b7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:57764/0 (socket says 172.21.15.70:57764) 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.562+0000 7efd8bfff640 1 -- 172.21.15.70:0/723495302 learned_addr learned my addr 172.21.15.70:0/723495302 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8b7fe640 1 -- 172.21.15.70:0/723495302 >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 msgr2=0x7efd8c15fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8b7fe640 1 --2- 172.21.15.70:0/723495302 >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 0x7efd8c15fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8b7fe640 1 -- 172.21.15.70:0/723495302 >> v1:172.21.15.96:6789/0 conn(0x7efd8c09adb0 legacy=0x7efd8c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8a7fc640 1 -- 172.21.15.70:0/723495302 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3182067144 0 0) 0x7efd8c070720 con 0x7efd8c09adb0 2024-04-15T15:15:48.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8b7fe640 1 -- 172.21.15.70:0/723495302 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c070a90 con 0x7efd8c09a9d0 2024-04-15T15:15:48.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.563+0000 7efd8affd640 1 --2- 172.21.15.70:0/723495302 >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 0x7efd8c15fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd8b7fe640 1 --2- 172.21.15.70:0/723495302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097d0 tx=0x7efd7c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5075750a5cd98daa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd8a7fc640 1 -- 172.21.15.70:0/723495302 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c035020 con 0x7efd8c09a9d0 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd8a7fc640 1 -- 172.21.15.70:0/723495302 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd7c031800 con 0x7efd8c09a9d0 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd8a7fc640 1 -- 172.21.15.70:0/723495302 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c031bd0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 -- 172.21.15.70:0/723495302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 msgr2=0x7efd8c160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 --2- 172.21.15.70:0/723495302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0097d0 tx=0x7efd7c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 -- 172.21.15.70:0/723495302 shutdown_connections 2024-04-15T15:15:48.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 --2- 172.21.15.70:0/723495302 >> v2:172.21.15.96:3300/0 conn(0x7efd8c0701f0 0x7efd8c15fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 --2- 172.21.15.70:0/723495302 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09a9d0 0x7efd8c160420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 -- 172.21.15.70:0/723495302 >> 172.21.15.70:0/723495302 conn(0x7efd8c0d2460 msgr2=0x7efd8c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 -- 172.21.15.70:0/723495302 shutdown_connections 2024-04-15T15:15:48.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.564+0000 7efd91de2640 1 -- 172.21.15.70:0/723495302 wait complete. 2024-04-15T15:15:48.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.566+0000 7efd91de2640 1 Processor -- start 2024-04-15T15:15:48.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.566+0000 7efd91de2640 1 -- start start 2024-04-15T15:15:48.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.566+0000 7efd91de2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd8c0701f0 0x7efd8c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09adb0 0x7efd8c14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c071050 con 0x7efd8c09a9d0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd8c0d08c0 con 0x7efd8c0701f0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efd8c0b1c60 con 0x7efd8c09adb0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45412/0 (socket says 172.21.15.70:45412) 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 -- 172.21.15.70:0/516836907 learned_addr learned my addr 172.21.15.70:0/516836907 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 -- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09adb0 msgr2=0x7efd8c14d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8bfff640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09adb0 0x7efd8c14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8b7fe640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd8c0701f0 0x7efd8c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09adb0 0x7efd8c14d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 -- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd8c0701f0 msgr2=0x7efd8c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd8c0701f0 0x7efd8c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 -- 172.21.15.70:0/516836907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd8c0ce130 con 0x7efd8c09a9d0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd8affd640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e930 tx=0x7efd8000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd80017070 con 0x7efd8c09a9d0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd8000c9a0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd8c0ce3d0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd8000cd90 con 0x7efd8c09a9d0 2024-04-15T15:15:48.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.567+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd8c14dd80 con 0x7efd8c09a9d0 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.568+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd8c0ce7c0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.569+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7efd8001a070 con 0x7efd8c09a9d0 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.569+0000 7efd73fff640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 0x7efd7806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.569+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7efd8008b9c0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.569+0000 7efd8b7fe640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 0x7efd7806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.570+0000 7efd8b7fe640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 0x7efd7806edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd7c036000 tx=0x7efd7c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.574+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd8008de20 con 0x7efd8c09a9d0 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.748+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efd8c0689e0 con 0x7efd8c09a9d0 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.748+0000 7efd73fff640 1 -- 172.21.15.70:0/516836907 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7efd80059a30 con 0x7efd8c09a9d0 2024-04-15T15:15:48.750 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 msgr2=0x7efd7806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 0x7efd7806edb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efd7c036000 tx=0x7efd7c008030 comp rx=0 tx=0).stop 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 msgr2=0x7efd8c0af640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7efd8000e930 tx=0x7efd8000edf0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 shutdown_connections 2024-04-15T15:15:48.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efd8c09adb0 0x7efd8c14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd7806c8f0 0x7efd7806edb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd8c09a9d0 0x7efd8c0af640 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 --2- 172.21.15.70:0/516836907 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd8c0701f0 0x7efd8c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 >> 172.21.15.70:0/516836907 conn(0x7efd8c0d2460 msgr2=0x7efd8c0ceee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 shutdown_connections 2024-04-15T15:15:48.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.750+0000 7efd91de2640 1 -- 172.21.15.70:0/516836907 wait complete. 2024-04-15T15:15:48.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:48.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.866+0000 7fe78ebb5640 1 Processor -- start 2024-04-15T15:15:48.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.866+0000 7fe78ebb5640 1 -- start start 2024-04-15T15:15:48.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe78ebb5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788070940 0x7fe78809b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe78ebb5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe788160310 0x7fe788070d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe78ebb5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7880d05e0 con 0x7fe7880d0da0 2024-04-15T15:15:48.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe78ebb5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe788071260 con 0x7fe788160310 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe78ebb5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe7880d1940 con 0x7fe788070940 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe787fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe788160310 0x7fe788070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe787fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe788160310 0x7fe788070d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45416/0 (socket says 172.21.15.70:45416) 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.867+0000 7fe787fff640 1 -- 172.21.15.70:0/1336173183 learned_addr learned my addr 172.21.15.70:0/1336173183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.868+0000 7fe787fff640 1 -- 172.21.15.70:0/1336173183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788070940 msgr2=0x7fe78809b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.868+0000 7fe787fff640 1 --2- 172.21.15.70:0/1336173183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788070940 0x7fe78809b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.868+0000 7fe787fff640 1 -- 172.21.15.70:0/1336173183 >> v1:172.21.15.96:6789/0 conn(0x7fe7880d0da0 legacy=0x7fe7880d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.868+0000 7fe787fff640 1 -- 172.21.15.70:0/1336173183 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78815f720 con 0x7fe788160310 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.868+0000 7fe787fff640 1 --2- 172.21.15.70:0/1336173183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe788160310 0x7fe788070d20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009870 tx=0x7fe77c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afa97d66022c32dc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe7877fe640 1 -- 172.21.15.70:0/1336173183 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c035020 con 0x7fe788160310 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe7877fe640 1 -- 172.21.15.70:0/1336173183 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe77c031640 con 0x7fe788160310 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe7877fe640 1 -- 172.21.15.70:0/1336173183 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c02d040 con 0x7fe788160310 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 -- 172.21.15.70:0/1336173183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe788160310 msgr2=0x7fe788070d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/1336173183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe788160310 0x7fe788070d20 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe77c009870 tx=0x7fe77c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 -- 172.21.15.70:0/1336173183 shutdown_connections 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/1336173183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe788160310 0x7fe788070d20 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/1336173183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788070940 0x7fe78809b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 -- 172.21.15.70:0/1336173183 >> 172.21.15.70:0/1336173183 conn(0x7fe7880b3f40 msgr2=0x7fe7880b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 -- 172.21.15.70:0/1336173183 shutdown_connections 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.869+0000 7fe78ebb5640 1 -- 172.21.15.70:0/1336173183 wait complete. 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.870+0000 7fe78ebb5640 1 Processor -- start 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.870+0000 7fe78ebb5640 1 -- start start 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 0x7fe7880c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788160310 0x7fe78809d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe7880d0880 con 0x7fe7880d0da0 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe78815fce0 con 0x7fe788070940 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe78ebb5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe788160770 con 0x7fe788160310 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe787fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe787fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45422/0 (socket says 172.21.15.70:45422) 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.871+0000 7fe787fff640 1 -- 172.21.15.70:0/343255593 learned_addr learned my addr 172.21.15.70:0/343255593 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 -- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788160310 msgr2=0x7fe78809d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788160310 0x7fe78809d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 -- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 msgr2=0x7fe7880c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe78c951640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 0x7fe7880c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 0x7fe7880c69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:48.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 -- 172.21.15.70:0/343255593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7881f5640 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe78c951640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 0x7fe7880c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe787fff640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe77c000c00 tx=0x7fe77c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c03ac60 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe77c03ae20 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7880c5ed0 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c044630 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.872+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7881f5920 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.873+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7880689e0 con 0x7fe7880d0da0 2024-04-15T15:15:48.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.874+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe77c038020 con 0x7fe7880d0da0 2024-04-15T15:15:48.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.874+0000 7fe7857fa640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 0x7fe77406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:48.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.874+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe77c0adc20 con 0x7fe7880d0da0 2024-04-15T15:15:48.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.874+0000 7fe78c951640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 0x7fe77406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:48.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.874+0000 7fe78c951640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 0x7fe77406edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe778007eb0 tx=0x7fe7780073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:48.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:48.878+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe77c03f070 con 0x7fe7880d0da0 2024-04-15T15:15:49.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.053+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe788070720 con 0x7fe7880d0da0 2024-04-15T15:15:49.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.054+0000 7fe7857fa640 1 -- 172.21.15.70:0/343255593 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fe77c07bda0 con 0x7fe7880d0da0 2024-04-15T15:15:49.056 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 msgr2=0x7fe77406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 0x7fe77406edb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe778007eb0 tx=0x7fe7780073c0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 msgr2=0x7fe7880b6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe77c000c00 tx=0x7fe77c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 shutdown_connections 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe788160310 0x7fe78809d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe77406c8f0 0x7fe77406edb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe7880d0da0 0x7fe7880b6c10 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 --2- 172.21.15.70:0/343255593 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe788070940 0x7fe7880c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 >> 172.21.15.70:0/343255593 conn(0x7fe7880b3f40 msgr2=0x7fe7880d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 shutdown_connections 2024-04-15T15:15:49.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.056+0000 7fe78ebb5640 1 -- 172.21.15.70:0/343255593 wait complete. 2024-04-15T15:15:49.069 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:49.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.169+0000 7f6749109640 1 Processor -- start 2024-04-15T15:15:49.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.169+0000 7f6749109640 1 -- start start 2024-04-15T15:15:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6749109640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67440d0690 0x7f67440d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6749109640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 0x7f674415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6749109640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67440d0fb0 con 0x7f674415ef70 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6749109640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f674415cb50 con 0x7f67440d0690 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6749109640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f674415bfd0 con 0x7f67440b3160 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.170+0000 7f6742d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67440d0690 0x7f67440d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f674359e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f674415ef70 0x7f674415f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55884/0 (socket says 172.21.15.70:55884) 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6742d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67440d0690 0x7f67440d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45424/0 (socket says 172.21.15.70:45424) 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f674359e640 1 -- 172.21.15.70:0/1265967591 learned_addr learned my addr 172.21.15.70:0/1265967591 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f674259c640 1 --2- 172.21.15.70:0/1265967591 >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 0x7f674415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6742d9d640 1 -- 172.21.15.70:0/1265967591 >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 msgr2=0x7f674415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6742d9d640 1 --2- 172.21.15.70:0/1265967591 >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 0x7f674415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6742d9d640 1 -- 172.21.15.70:0/1265967591 >> v1:172.21.15.70:6789/0 conn(0x7f674415ef70 legacy=0x7f674415f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6742d9d640 1 -- 172.21.15.70:0/1265967591 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674415c2b0 con 0x7f67440d0690 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f6741d9b640 1 -- 172.21.15.70:0/1265967591 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3654776747 0 0) 0x7f67440d0fb0 con 0x7f674415ef70 2024-04-15T15:15:49.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.171+0000 7f674259c640 1 --2- 172.21.15.70:0/1265967591 >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 0x7f674415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6742d9d640 1 --2- 172.21.15.70:0/1265967591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f67440d0a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f674406fec0 tx=0x7f673c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172ec77d239b3c36 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6741d9b640 1 -- 172.21.15.70:0/1265967591 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c039070 con 0x7f67440d0690 2024-04-15T15:15:49.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6741d9b640 1 -- 172.21.15.70:0/1265967591 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f673c02cc80 con 0x7f67440d0690 2024-04-15T15:15:49.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6741d9b640 1 -- 172.21.15.70:0/1265967591 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c034660 con 0x7f67440d0690 2024-04-15T15:15:49.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 -- 172.21.15.70:0/1265967591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 msgr2=0x7f67440d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 --2- 172.21.15.70:0/1265967591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f67440d0a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f674406fec0 tx=0x7f673c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 -- 172.21.15.70:0/1265967591 shutdown_connections 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 --2- 172.21.15.70:0/1265967591 >> v2:172.21.15.96:3300/0 conn(0x7f67440b3160 0x7f674415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 --2- 172.21.15.70:0/1265967591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f67440d0a70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 -- 172.21.15.70:0/1265967591 >> 172.21.15.70:0/1265967591 conn(0x7f674406be00 msgr2=0x7f674406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 -- 172.21.15.70:0/1265967591 shutdown_connections 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.172+0000 7f6749109640 1 -- 172.21.15.70:0/1265967591 wait complete. 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 Processor -- start 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 -- start start 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67440b3160 0x7f674409b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f674415ef70 0x7f67440cf3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67440b4fb0 con 0x7f67440d0690 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f674415c870 con 0x7f674415ef70 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.174+0000 7f6749109640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f674415bbd0 con 0x7f67440b3160 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45428/0 (socket says 172.21.15.70:45428) 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 -- 172.21.15.70:0/326299867 learned_addr learned my addr 172.21.15.70:0/326299867 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 -- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67440b3160 msgr2=0x7f674409b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67440b3160 0x7f674409b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 -- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f674415ef70 msgr2=0x7f67440cf3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f674415ef70 0x7f67440cf3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 -- 172.21.15.70:0/326299867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674409a360 con 0x7f67440d0690 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f674259c640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d800 tx=0x7f672c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c004030 con 0x7f67440d0690 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f672c0041d0 con 0x7f67440d0690 2024-04-15T15:15:49.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674409a640 con 0x7f67440d0690 2024-04-15T15:15:49.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f672c0096a0 con 0x7f67440d0690 2024-04-15T15:15:49.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.175+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674409ac00 con 0x7f67440d0690 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.176+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67440689e0 con 0x7f67440d0690 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.177+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f672c009840 con 0x7f67440d0690 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.177+0000 7f673b7fe640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 0x7f672406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.177+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f672c08c1b0 con 0x7f67440d0690 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.177+0000 7f6742d9d640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 0x7f672406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.177+0000 7f6742d9d640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 0x7f672406edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f673c036000 tx=0x7f673c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.181+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f672c05a210 con 0x7f67440d0690 2024-04-15T15:15:49.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.361+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6744072d00 con 0x7f67440d0690 2024-04-15T15:15:49.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.362+0000 7f673b7fe640 1 -- 172.21.15.70:0/326299867 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18454 (secure 0 0 0) 0x7f672c059b10 con 0x7f67440d0690 2024-04-15T15:15:49.365 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:49.365 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":25,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:14.672468+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 msgr2=0x7f672406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 0x7f672406edb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f673c036000 tx=0x7f673c02e000 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 msgr2=0x7f674414c530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f672c00d800 tx=0x7f672c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 shutdown_connections 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f674415ef70 0x7f67440cf3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f672406c8f0 0x7f672406edb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67440d0690 0x7f674414c530 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 --2- 172.21.15.70:0/326299867 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67440b3160 0x7f674409b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 >> 172.21.15.70:0/326299867 conn(0x7f674406be00 msgr2=0x7f67440d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 shutdown_connections 2024-04-15T15:15:49.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.364+0000 7f6749109640 1 -- 172.21.15.70:0/326299867 wait complete. 2024-04-15T15:15:49.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:49.378 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:49.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.482+0000 7f8ab2c35640 1 Processor -- start 2024-04-15T15:15:49.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.482+0000 7f8ab2c35640 1 -- start start 2024-04-15T15:15:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab2c35640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8aac071d10 0x7f8aac0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab2c35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac0d0bd0 0x7f8aac06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab2c35640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aac15db40 con 0x7f8aac072d80 2024-04-15T15:15:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab2c35640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8aac072170 con 0x7f8aac071d10 2024-04-15T15:15:49.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab2c35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac15cfc0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8aac071d10 0x7f8aac0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8aac071d10 0x7f8aac0d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45436/0 (socket says 172.21.15.70:45436) 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 -- 172.21.15.70:0/4034226885 learned_addr learned my addr 172.21.15.70:0/4034226885 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8ab11d2640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac0d0bd0 0x7f8aac06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 -- 172.21.15.70:0/4034226885 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac0d0bd0 msgr2=0x7f8aac06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac0d0bd0 0x7f8aac06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 -- 172.21.15.70:0/4034226885 >> v1:172.21.15.96:6789/0 conn(0x7f8aac072d80 legacy=0x7f8aac0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.483+0000 7f8aa3fff640 1 -- 172.21.15.70:0/4034226885 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac15d2a0 con 0x7f8aac071d10 2024-04-15T15:15:49.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.484+0000 7f8aa3fff640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac071d10 0x7f8aac0d0690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009870 tx=0x7f8a9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5267faff07aaf78 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.484+0000 7f8aa37fe640 1 -- 172.21.15.70:0/4034226885 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c035020 con 0x7f8aac071d10 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.484+0000 7f8aa37fe640 1 -- 172.21.15.70:0/4034226885 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9c031640 con 0x7f8aac071d10 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.484+0000 7f8aa37fe640 1 -- 172.21.15.70:0/4034226885 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c02d040 con 0x7f8aac071d10 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 -- 172.21.15.70:0/4034226885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac071d10 msgr2=0x7f8aac0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac071d10 0x7f8aac0d0690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009870 tx=0x7f8a9c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 -- 172.21.15.70:0/4034226885 shutdown_connections 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac0d0bd0 0x7f8aac06e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/4034226885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac071d10 0x7f8aac0d0690 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 -- 172.21.15.70:0/4034226885 >> 172.21.15.70:0/4034226885 conn(0x7f8aac06be00 msgr2=0x7f8aac06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 -- 172.21.15.70:0/4034226885 shutdown_connections 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.485+0000 7f8ab2c35640 1 -- 172.21.15.70:0/4034226885 wait complete. 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.486+0000 7f8ab2c35640 1 Processor -- start 2024-04-15T15:15:49.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.486+0000 7f8ab2c35640 1 -- start start 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac071d10 0x7f8aac157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 0x7f8aac157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac0afaf0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8aac15d860 con 0x7f8aac072d80 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab2c35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8aac15cb90 con 0x7f8aac071d10 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45440/0 (socket says 172.21.15.70:45440) 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 -- 172.21.15.70:0/388751222 learned_addr learned my addr 172.21.15.70:0/388751222 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab09d1640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac071d10 0x7f8aac157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8aa3fff640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 0x7f8aac157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 -- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac071d10 msgr2=0x7f8aac157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac071d10 0x7f8aac157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 -- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 msgr2=0x7f8aac157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 0x7f8aac157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8ab11d2640 1 -- 172.21.15.70:0/388751222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aac1f2f40 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.487+0000 7f8aa3fff640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 0x7f8aac157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8ab11d2640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4007e40 tx=0x7f8aa400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4014620 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4019070 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aac0cce20 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4014a20 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.488+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aac0cd3e0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.489+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aac0689e0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.489+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8aa4014bc0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.490+0000 7f8aa17fa640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 0x7f8a9806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.490+0000 7f8ab09d1640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 0x7f8a9806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.490+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8aa408c700 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.490+0000 7f8ab09d1640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 0x7f8a9806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0ad640 tx=0x7f8aa8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.494+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8aa405a7f0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.671+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8aac0ccc00 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.671+0000 7f8aa17fa640 1 -- 172.21.15.70:0/388751222 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f8aa405a0f0 con 0x7f8aac0d0bd0 2024-04-15T15:15:49.673 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:49.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 msgr2=0x7f8a9806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 0x7f8a9806edb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8aac0ad640 tx=0x7f8aa8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:49.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 msgr2=0x7f8aac1f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4007e40 tx=0x7f8aa400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 shutdown_connections 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8a9806c8f0 0x7f8a9806edb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8aac0d0bd0 0x7f8aac1f21b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8aac072d80 0x7f8aac157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 --2- 172.21.15.70:0/388751222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8aac071d10 0x7f8aac157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 >> 172.21.15.70:0/388751222 conn(0x7f8aac06be00 msgr2=0x7f8aac0b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 shutdown_connections 2024-04-15T15:15:49.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.673+0000 7f8ab2c35640 1 -- 172.21.15.70:0/388751222 wait complete. 2024-04-15T15:15:49.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:49.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.791+0000 7fb198b59640 1 Processor -- start 2024-04-15T15:15:49.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.791+0000 7fb198b59640 1 -- start start 2024-04-15T15:15:49.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb198b59640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb194070940 0x7fb19409b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb198b59640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb1940d0da0 0x7fb1940d1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb198b59640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1940d05e0 con 0x7fb194160310 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb198b59640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb19415f720 con 0x7fb194070940 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb198b59640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb19415fa00 con 0x7fb1940d0da0 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb194070940 0x7fb19409b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb194070940 0x7fb19409b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45442/0 (socket says 172.21.15.70:45442) 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 -- 172.21.15.70:0/4078760770 learned_addr learned my addr 172.21.15.70:0/4078760770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb192d9d640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb1940d0da0 0x7fb1940d1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 -- 172.21.15.70:0/4078760770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb1940d0da0 msgr2=0x7fb1940d1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb1940d0da0 0x7fb1940d1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 -- 172.21.15.70:0/4078760770 >> v1:172.21.15.96:6789/0 conn(0x7fb194160310 legacy=0x7fb194070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.792+0000 7fb19259c640 1 -- 172.21.15.70:0/4078760770 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19415fce0 con 0x7fb194070940 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb19259c640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb194070940 0x7fb19409b880 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb184009870 tx=0x7fb18402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d05a84848c01de1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb191d9b640 1 -- 172.21.15.70:0/4078760770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184034070 con 0x7fb194070940 2024-04-15T15:15:49.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb191d9b640 1 -- 172.21.15.70:0/4078760770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb184030840 con 0x7fb194070940 2024-04-15T15:15:49.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb191d9b640 1 -- 172.21.15.70:0/4078760770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb184030c10 con 0x7fb194070940 2024-04-15T15:15:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb198b59640 1 -- 172.21.15.70:0/4078760770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb194070940 msgr2=0x7fb19409b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.793+0000 7fb198b59640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb194070940 0x7fb19409b880 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb184009870 tx=0x7fb18402b980 comp rx=0 tx=0).stop 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 -- 172.21.15.70:0/4078760770 shutdown_connections 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb1940d0da0 0x7fb1940d1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 --2- 172.21.15.70:0/4078760770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb194070940 0x7fb19409b880 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 -- 172.21.15.70:0/4078760770 >> 172.21.15.70:0/4078760770 conn(0x7fb1940b3f40 msgr2=0x7fb1940b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 -- 172.21.15.70:0/4078760770 shutdown_connections 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.794+0000 7fb198b59640 1 -- 172.21.15.70:0/4078760770 wait complete. 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.795+0000 7fb198b59640 1 Processor -- start 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.795+0000 7fb198b59640 1 -- start start 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 0x7fb19415aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb194160310 0x7fb1940c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1940d1770 con 0x7fb1940d0da0 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb194160770 con 0x7fb194070940 2024-04-15T15:15:49.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb1940d18f0 con 0x7fb194160310 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45452/0 (socket says 172.21.15.70:45452) 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 -- 172.21.15.70:0/1087751102 learned_addr learned my addr 172.21.15.70:0/1087751102 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb192d9d640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb194160310 0x7fb1940c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb19259c640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 0x7fb19415aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 -- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb194160310 msgr2=0x7fb1940c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb194160310 0x7fb1940c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 -- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 msgr2=0x7fb19415aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 0x7fb19415aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 -- 172.21.15.70:0/1087751102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1940c5ed0 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb19259c640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 0x7fb19415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18bfff640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00c400 tx=0x7fb17c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c0052d0 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17c005490 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17c007260 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1940c61b0 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.796+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1940c6770 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.797+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1940689e0 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.798+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb17c007400 con 0x7fb1940d0da0 2024-04-15T15:15:49.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.798+0000 7fb18affd640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 0x7fb17406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.798+0000 7fb19259c640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 0x7fb17406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.798+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb17c08b3a0 con 0x7fb1940d0da0 2024-04-15T15:15:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.799+0000 7fb19259c640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 0x7fb17406ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb184036000 tx=0x7fb18403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:49.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.802+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb17c059490 con 0x7fb1940d0da0 2024-04-15T15:15:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.976+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb19409c610 con 0x7fb1940d0da0 2024-04-15T15:15:49.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.977+0000 7fb18affd640 1 -- 172.21.15.70:0/1087751102 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7fb17c058d90 con 0x7fb1940d0da0 2024-04-15T15:15:49.978 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 msgr2=0x7fb17406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 0x7fb17406ede0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb184036000 tx=0x7fb18403b040 comp rx=0 tx=0).stop 2024-04-15T15:15:49.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 msgr2=0x7fb19409d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb17c00c400 tx=0x7fb17c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 shutdown_connections 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb194160310 0x7fb1940c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb17406c920 0x7fb17406ede0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1940d0da0 0x7fb19409d240 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 --2- 172.21.15.70:0/1087751102 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb194070940 0x7fb19415aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.978+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 >> 172.21.15.70:0/1087751102 conn(0x7fb1940b3f40 msgr2=0x7fb1940af1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.979+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 shutdown_connections 2024-04-15T15:15:49.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:49.979+0000 7fb198b59640 1 -- 172.21.15.70:0/1087751102 wait complete. 2024-04-15T15:15:49.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:50.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.097+0000 7fa87f00d640 1 Processor -- start 2024-04-15T15:15:50.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.097+0000 7fa87f00d640 1 -- start start 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8780d0690 0x7fa8780d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8780b3160 0x7fa87815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa87815ef70 0x7fa87815f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa8780d0fb0 con 0x7fa8780d0690 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa87815cb50 con 0x7fa87815ef70 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87f00d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa87815bfd0 con 0x7fa8780b3160 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8780d0690 0x7fa8780d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8780d0690 0x7fa8780d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45464/0 (socket says 172.21.15.70:45464) 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 -- 172.21.15.70:0/569214307 learned_addr learned my addr 172.21.15.70:0/569214307 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 -- 172.21.15.70:0/569214307 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8780b3160 msgr2=0x7fa87815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 --2- 172.21.15.70:0/569214307 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8780b3160 0x7fa87815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 -- 172.21.15.70:0/569214307 >> v2:172.21.15.96:3300/0 conn(0x7fa87815ef70 msgr2=0x7fa87815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87d5aa640 1 --2- 172.21.15.70:0/569214307 >> v2:172.21.15.96:3300/0 conn(0x7fa87815ef70 0x7fa87815f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 --2- 172.21.15.70:0/569214307 >> v2:172.21.15.96:3300/0 conn(0x7fa87815ef70 0x7fa87815f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.098+0000 7fa87cda9640 1 -- 172.21.15.70:0/569214307 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87815c2b0 con 0x7fa8780d0690 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.099+0000 7fa87cda9640 1 --2- 172.21.15.70:0/569214307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780d0690 0x7fa8780d0a70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa8780b4ba0 tx=0x7fa86c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a5e02659a68fb50 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.099+0000 7fa8777fe640 1 -- 172.21.15.70:0/569214307 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c035020 con 0x7fa8780d0690 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.099+0000 7fa8777fe640 1 -- 172.21.15.70:0/569214307 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c031870 con 0x7fa8780d0690 2024-04-15T15:15:50.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.099+0000 7fa8777fe640 1 -- 172.21.15.70:0/569214307 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c031c40 con 0x7fa8780d0690 2024-04-15T15:15:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 -- 172.21.15.70:0/569214307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780d0690 msgr2=0x7fa8780d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 --2- 172.21.15.70:0/569214307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780d0690 0x7fa8780d0a70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa8780b4ba0 tx=0x7fa86c02b910 comp rx=0 tx=0).stop 2024-04-15T15:15:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 -- 172.21.15.70:0/569214307 shutdown_connections 2024-04-15T15:15:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 --2- 172.21.15.70:0/569214307 >> v2:172.21.15.96:3300/0 conn(0x7fa87815ef70 0x7fa87815f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 --2- 172.21.15.70:0/569214307 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8780b3160 0x7fa87815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 --2- 172.21.15.70:0/569214307 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780d0690 0x7fa8780d0a70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 -- 172.21.15.70:0/569214307 >> 172.21.15.70:0/569214307 conn(0x7fa87806be00 msgr2=0x7fa87806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 -- 172.21.15.70:0/569214307 shutdown_connections 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.100+0000 7fa87f00d640 1 -- 172.21.15.70:0/569214307 wait complete. 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.101+0000 7fa87f00d640 1 Processor -- start 2024-04-15T15:15:50.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.101+0000 7fa87f00d640 1 -- start start 2024-04-15T15:15:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8780d0690 0x7fa878149bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa87815ef70 0x7fa878147230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8780b4fb0 con 0x7fa8780b3160 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa87815c870 con 0x7fa8780d0690 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87f00d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa87815bbd0 con 0x7fa87815ef70 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45480/0 (socket says 172.21.15.70:45480) 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 -- 172.21.15.70:0/3327569511 learned_addr learned my addr 172.21.15.70:0/3327569511 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87d5aa640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa87815ef70 0x7fa878147230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa877fff640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8780d0690 0x7fa878149bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 -- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa87815ef70 msgr2=0x7fa878147230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa87815ef70 0x7fa878147230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 -- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8780d0690 msgr2=0x7fa878149bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8780d0690 0x7fa878149bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 -- 172.21.15.70:0/3327569511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87814a310 con 0x7fa8780b3160 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa87cda9640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa86c000c00 tx=0x7fa86c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.102+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c040070 con 0x7fa8780b3160 2024-04-15T15:15:50.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.103+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86c03a420 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.103+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa87814a5f0 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.103+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86c035040 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.103+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa878147cd0 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.104+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8780689e0 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.104+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa86c04b060 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.104+0000 7fa8757fa640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.104+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa86c0add60 con 0x7fa8780b3160 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.104+0000 7fa877fff640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.105+0000 7fa877fff640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 0x7fa85c06edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8781f5ba0 tx=0x7fa864008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.109+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa86c07bee0 con 0x7fa8780b3160 2024-04-15T15:15:50.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.284+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa8780739e0 con 0x7fa8780b3160 2024-04-15T15:15:50.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.284+0000 7fa8757fa640 1 -- 172.21.15.70:0/3327569511 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fa86c07b7e0 con 0x7fa8780b3160 2024-04-15T15:15:50.286 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:50.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.286+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 msgr2=0x7fa85c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.286+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 0x7fa85c06edb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa8781f5ba0 tx=0x7fa864008040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.286+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 msgr2=0x7fa878158c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.286+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa86c000c00 tx=0x7fa86c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 shutdown_connections 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa87815ef70 0x7fa878147230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8780d0690 0x7fa878149bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa85c06c8f0 0x7fa85c06edb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 --2- 172.21.15.70:0/3327569511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8780b3160 0x7fa878158c80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 >> 172.21.15.70:0/3327569511 conn(0x7fa87806be00 msgr2=0x7fa87815a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 shutdown_connections 2024-04-15T15:15:50.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.287+0000 7fa87f00d640 1 -- 172.21.15.70:0/3327569511 wait complete. 2024-04-15T15:15:50.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:50.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.404+0000 7f202e51c640 1 Processor -- start 2024-04-15T15:15:50.406 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.404+0000 7f202e51c640 1 -- start start 2024-04-15T15:15:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f202e51c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f20280d3270 0x7f20280cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f202e51c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20280b47a0 con 0x7f20280d2360 2024-04-15T15:15:50.407 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f202e51c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20280b4a80 con 0x7f202815e700 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f202e51c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f20280b53b0 con 0x7f20280d3270 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f202cab9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f20280d3270 0x7f20280cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f201ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f20280d2360 0x7f20280d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55890/0 (socket says 172.21.15.70:55890) 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.406+0000 7f201ffff640 1 -- 172.21.15.70:0/3744288353 learned_addr learned my addr 172.21.15.70:0/3744288353 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.407+0000 7f20277fe640 1 -- 172.21.15.70:0/3744288353 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1381589382 0 0) 0x7f20280b47a0 con 0x7f20280d2360 2024-04-15T15:15:50.408 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.407+0000 7f20277fe640 1 -- 172.21.15.70:0/3744288353 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2014003520 con 0x7f20280d2360 2024-04-15T15:15:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.407+0000 7f202cab9640 1 -- 172.21.15.70:0/3744288353 >> v1:172.21.15.96:6789/0 conn(0x7f202815e700 legacy=0x7f20280b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.407+0000 7f202cab9640 1 -- 172.21.15.70:0/3744288353 >> v1:172.21.15.70:6789/0 conn(0x7f20280d2360 legacy=0x7f20280d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.409 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.407+0000 7f202cab9640 1 -- 172.21.15.70:0/3744288353 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20280b4d60 con 0x7f20280d3270 2024-04-15T15:15:50.411 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.410+0000 7f202cab9640 1 --2- 172.21.15.70:0/3744288353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280cf370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f20280b5530 tx=0x7f202002bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9169ee823dae19ed server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.410+0000 7f20277fe640 1 -- 172.21.15.70:0/3744288353 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020039070 con 0x7f20280d3270 2024-04-15T15:15:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.410+0000 7f20277fe640 1 -- 172.21.15.70:0/3744288353 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f202002cce0 con 0x7f20280d3270 2024-04-15T15:15:50.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.410+0000 7f20277fe640 1 -- 172.21.15.70:0/3744288353 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020034660 con 0x7f20280d3270 2024-04-15T15:15:50.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.411+0000 7f202e51c640 1 -- 172.21.15.70:0/3744288353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 msgr2=0x7f20280cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.411+0000 7f202e51c640 1 --2- 172.21.15.70:0/3744288353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280cf370 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f20280b5530 tx=0x7f202002bf30 comp rx=0 tx=0).stop 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.411+0000 7f202e51c640 1 -- 172.21.15.70:0/3744288353 shutdown_connections 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.411+0000 7f202e51c640 1 --2- 172.21.15.70:0/3744288353 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280cf370 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.411+0000 7f202e51c640 1 -- 172.21.15.70:0/3744288353 >> 172.21.15.70:0/3744288353 conn(0x7f202808bf60 msgr2=0x7f20280d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.412+0000 7f202e51c640 1 -- 172.21.15.70:0/3744288353 shutdown_connections 2024-04-15T15:15:50.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.412+0000 7f202e51c640 1 -- 172.21.15.70:0/3744288353 wait complete. 2024-04-15T15:15:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.414+0000 7f202e51c640 1 Processor -- start 2024-04-15T15:15:50.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.414+0000 7f202e51c640 1 -- start start 2024-04-15T15:15:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20280d2360 0x7f20280ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280d0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20280d1d40 con 0x7f202815e700 2024-04-15T15:15:50.418 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20280b4450 con 0x7f20280d3270 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202e51c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f20280d1ec0 con 0x7f20280d2360 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45488/0 (socket says 172.21.15.70:45488) 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 -- 172.21.15.70:0/2871731147 learned_addr learned my addr 172.21.15.70:0/2871731147 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f2027fff640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20280d2360 0x7f20280ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f201ffff640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 -- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20280d2360 msgr2=0x7f20280ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20280d2360 0x7f20280ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 -- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 msgr2=0x7f20280d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.415+0000 7f202cab9640 1 -- 172.21.15.70:0/2871731147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2028160870 con 0x7f202815e700 2024-04-15T15:15:50.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f202cab9640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2020009e80 tx=0x7f2020004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020039050 con 0x7f202815e700 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2020045070 con 0x7f202815e700 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f202814b710 con 0x7f202815e700 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2020009450 con 0x7f202815e700 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.416+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f202814e1d0 con 0x7f202815e700 2024-04-15T15:15:50.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.417+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20280d1660 con 0x7f202815e700 2024-04-15T15:15:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.417+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f202003e850 con 0x7f202815e700 2024-04-15T15:15:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.417+0000 7f20257fa640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 0x7f200806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.418+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f20200aeb90 con 0x7f202815e700 2024-04-15T15:15:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.418+0000 7f2027fff640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 0x7f200806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.418+0000 7f2027fff640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 0x7f200806edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f20180016d0 tx=0x7f2018009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.422+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20280d1660 con 0x7f202815e700 2024-04-15T15:15:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.595+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f20280689e0 con 0x7f202815e700 2024-04-15T15:15:50.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.596+0000 7f20257fa640 1 -- 172.21.15.70:0/2871731147 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) ==== 103+0+11 (secure 0 0 0) 0x7f2028160870 con 0x7f202815e700 2024-04-15T15:15:50.598 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:50.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 msgr2=0x7f200806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 0x7f200806edb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f20180016d0 tx=0x7f2018009040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 msgr2=0x7f20280b9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f2020009e80 tx=0x7f2020004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 shutdown_connections 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f200806c8f0 0x7f200806edb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f202815e700 0x7f20280b9350 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20280d3270 0x7f20280d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 --2- 172.21.15.70:0/2871731147 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20280d2360 0x7f20280ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 >> 172.21.15.70:0/2871731147 conn(0x7f202808bf60 msgr2=0x7f20280b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 shutdown_connections 2024-04-15T15:15:50.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.598+0000 7f202e51c640 1 -- 172.21.15.70:0/2871731147 wait complete. 2024-04-15T15:15:50.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.710+0000 7f31a8660640 1 Processor -- start 2024-04-15T15:15:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.710+0000 7f31a8660640 1 -- start start 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f31a0158770 0x7f31a015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a00b5230 0x7f31a0072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31a0072c60 0x7f31a015e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f31a00b4da0 con 0x7f31a0072c60 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f31a015c1d0 con 0x7f31a0158770 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.711+0000 7f31a8660640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f31a00b4190 con 0x7f31a00b5230 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31a0072c60 0x7f31a015e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f31a0072c60 0x7f31a015e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45502/0 (socket says 172.21.15.70:45502) 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2031234002 learned_addr learned my addr 172.21.15.70:0/2031234002 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a63fc640 1 --2- 172.21.15.70:0/2031234002 >> v2:172.21.15.96:3300/0 conn(0x7f31a0158770 0x7f31a015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a5bfb640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a00b5230 0x7f31a0072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2031234002 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a00b5230 msgr2=0x7f31a0072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a00b5230 0x7f31a0072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2031234002 >> v2:172.21.15.96:3300/0 conn(0x7f31a0158770 msgr2=0x7f31a015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2031234002 >> v2:172.21.15.96:3300/0 conn(0x7f31a0158770 0x7f31a015bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.712+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2031234002 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a00b4470 con 0x7f31a0072c60 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0072c60 0x7f31a015e800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f31a006fe80 tx=0x7f319802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=921793dbcb7cb2fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a53fa640 1 -- 172.21.15.70:0/2031234002 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f319802ca70 con 0x7f31a0072c60 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a53fa640 1 -- 172.21.15.70:0/2031234002 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f319802cc10 con 0x7f31a0072c60 2024-04-15T15:15:50.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a53fa640 1 -- 172.21.15.70:0/2031234002 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31980343f0 con 0x7f31a0072c60 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a8660640 1 -- 172.21.15.70:0/2031234002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0072c60 msgr2=0x7f31a015e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.713+0000 7f31a8660640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0072c60 0x7f31a015e800 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f31a006fe80 tx=0x7f319802bed0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 -- 172.21.15.70:0/2031234002 shutdown_connections 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0072c60 0x7f31a015e800 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 --2- 172.21.15.70:0/2031234002 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a00b5230 0x7f31a0072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 --2- 172.21.15.70:0/2031234002 >> v2:172.21.15.96:3300/0 conn(0x7f31a0158770 0x7f31a015bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 -- 172.21.15.70:0/2031234002 >> 172.21.15.70:0/2031234002 conn(0x7f31a006be00 msgr2=0x7f31a006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 -- 172.21.15.70:0/2031234002 shutdown_connections 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.714+0000 7f31a8660640 1 -- 172.21.15.70:0/2031234002 wait complete. 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 Processor -- start 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 -- start start 2024-04-15T15:15:50.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a0072c60 0x7f31a014d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5230 0x7f31a00cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f31a0071fa0 con 0x7f31a0158770 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f31a00b4a30 con 0x7f31a00b5230 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.715+0000 7f31a8660640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f31a0158b50 con 0x7f31a0072c60 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45508/0 (socket says 172.21.15.70:45508) 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2053514622 learned_addr learned my addr 172.21.15.70:0/2053514622 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a5bfb640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5230 0x7f31a00cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a63fc640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a0072c60 0x7f31a014d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a0072c60 msgr2=0x7f31a014d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a0072c60 0x7f31a014d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5230 msgr2=0x7f31a00cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5230 0x7f31a00cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 -- 172.21.15.70:0/2053514622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a00ae510 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a6bfd640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3198036040 tx=0x7f3198004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198043020 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f319803e450 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a00ae7f0 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3198009510 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.716+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a014dcb0 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.717+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31a00cc570 con 0x7f31a0158770 2024-04-15T15:15:50.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.718+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3198046020 con 0x7f31a0158770 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.718+0000 7f3192ffd640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.718+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f31980ae810 con 0x7f31a0158770 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.718+0000 7f31a63fc640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:50.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.718+0000 7f31a63fc640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 0x7f318406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:50.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.722+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f319804d030 con 0x7f31a0158770 2024-04-15T15:15:50.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.896+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f31a00689e0 con 0x7f31a0158770 2024-04-15T15:15:50.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.898+0000 7f3192ffd640 1 -- 172.21.15.70:0/2053514622 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18842 (secure 0 0 0) 0x7f319807c8d0 con 0x7f31a0158770 2024-04-15T15:15:50.901 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:50.901 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":26,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:50.402311+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 msgr2=0x7f318406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 0x7f318406edb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3194006fd0 tx=0x7f3194008040 comp rx=0 tx=0).stop 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 msgr2=0x7f31a015cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3198036040 tx=0x7f3198004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 shutdown_connections 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f318406c8f0 0x7f318406edb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f31a0158770 0x7f31a015cb00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f31a00b5230 0x7f31a00cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 --2- 172.21.15.70:0/2053514622 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f31a0072c60 0x7f31a014d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 >> 172.21.15.70:0/2053514622 conn(0x7f31a006be00 msgr2=0x7f31a00b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 shutdown_connections 2024-04-15T15:15:50.904 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:50.900+0000 7f31a8660640 1 -- 172.21.15.70:0/2053514622 wait complete. 2024-04-15T15:15:50.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:50.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.017+0000 7f8d2ccba640 1 Processor -- start 2024-04-15T15:15:51.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.017+0000 7f8d2ccba640 1 -- start start 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d2815c8d0 0x7f8d280cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d2809b650 0x7f8d2809a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8d2809af10 0x7f8d280d0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8d280b5b00 con 0x7f8d2809b650 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8d280d0350 con 0x7f8d2809af10 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2ccba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8d280ce100 con 0x7f8d2815c8d0 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d2809b650 0x7f8d2809a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8d2809b650 0x7f8d2809a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45520/0 (socket says 172.21.15.70:45520) 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 -- 172.21.15.70:0/1361595951 learned_addr learned my addr 172.21.15.70:0/1361595951 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d2659c640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d2815c8d0 0x7f8d280cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d26d9d640 1 --2- 172.21.15.70:0/1361595951 >> v2:172.21.15.96:3300/0 conn(0x7f8d2809af10 0x7f8d280d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 -- 172.21.15.70:0/1361595951 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d2815c8d0 msgr2=0x7f8d280cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d2815c8d0 0x7f8d280cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 -- 172.21.15.70:0/1361595951 >> v2:172.21.15.96:3300/0 conn(0x7f8d2809af10 msgr2=0x7f8d280d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/1361595951 >> v2:172.21.15.96:3300/0 conn(0x7f8d2809af10 0x7f8d280d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.018+0000 7f8d1ffff640 1 -- 172.21.15.70:0/1361595951 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d280b54b0 con 0x7f8d2809b650 2024-04-15T15:15:51.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.019+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809b650 0x7f8d2809a9d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8d280c5c40 tx=0x7f8d1002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe573373248cd0ea server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.019+0000 7f8d25d9b640 1 -- 172.21.15.70:0/1361595951 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1002e070 con 0x7f8d2809b650 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.019+0000 7f8d25d9b640 1 -- 172.21.15.70:0/1361595951 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d10037070 con 0x7f8d2809b650 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.019+0000 7f8d25d9b640 1 -- 172.21.15.70:0/1361595951 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10031a00 con 0x7f8d2809b650 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 -- 172.21.15.70:0/1361595951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809b650 msgr2=0x7f8d2809a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809b650 0x7f8d2809a9d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8d280c5c40 tx=0x7f8d1002b880 comp rx=0 tx=0).stop 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 -- 172.21.15.70:0/1361595951 shutdown_connections 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/1361595951 >> v2:172.21.15.96:3300/0 conn(0x7f8d2809af10 0x7f8d280d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809b650 0x7f8d2809a9d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/1361595951 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d2815c8d0 0x7f8d280cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 -- 172.21.15.70:0/1361595951 >> 172.21.15.70:0/1361595951 conn(0x7f8d280b3fb0 msgr2=0x7f8d280b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 -- 172.21.15.70:0/1361595951 shutdown_connections 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.020+0000 7f8d2ccba640 1 -- 172.21.15.70:0/1361595951 wait complete. 2024-04-15T15:15:51.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.021+0000 7f8d2ccba640 1 Processor -- start 2024-04-15T15:15:51.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.021+0000 7f8d2ccba640 1 -- start start 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 0x7f8d28155c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 0x7f8d28157800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d280d0c00 con 0x7f8d2809af10 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8d280d0a80 con 0x7f8d2809b650 2024-04-15T15:15:51.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8d280d0d80 con 0x7f8d28155520 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45528/0 (socket says 172.21.15.70:45528) 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 -- 172.21.15.70:0/2620640559 learned_addr learned my addr 172.21.15.70:0/2620640559 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 0x7f8d28155c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d26d9d640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 0x7f8d28157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 -- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 msgr2=0x7f8d28157800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 0x7f8d28157800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 -- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 msgr2=0x7f8d28155c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 0x7f8d28155c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 -- 172.21.15.70:0/2620640559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d28156c70 con 0x7f8d2809af10 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d26d9d640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 0x7f8d28157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 0x7f8d28155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2659c640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e870 tx=0x7f8d1800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d18017070 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d1800c940 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d281563d0 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.022+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1800cd30 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.023+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d28156990 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.024+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d28067af0 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.024+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8d1801a070 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.025+0000 7f8d1effd640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 0x7f8d0806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.025+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 0x7f8d0806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.025+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f8d1808bd90 con 0x7f8d2809af10 2024-04-15T15:15:51.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.025+0000 7f8d1ffff640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 0x7f8d0806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d10036000 tx=0x7f8d10008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.029+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8d28067af0 con 0x7f8d2809af10 2024-04-15T15:15:51.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.203+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d280690f0 con 0x7f8d2809af10 2024-04-15T15:15:51.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.203+0000 7f8d1effd640 1 -- 172.21.15.70:0/2620640559 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) ==== 93+0+10 (secure 0 0 0) 0x7f8d18059dc0 con 0x7f8d2809af10 2024-04-15T15:15:51.207 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:51.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 msgr2=0x7f8d0806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 0x7f8d0806edb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f8d10036000 tx=0x7f8d10008030 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 msgr2=0x7f8d2815aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e870 tx=0x7f8d1800ed30 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 shutdown_connections 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8d28155520 0x7f8d28157800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8d2809b650 0x7f8d28155c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8d0806c8f0 0x7f8d0806edb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 --2- 172.21.15.70:0/2620640559 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d2809af10 0x7f8d2815aab0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 >> 172.21.15.70:0/2620640559 conn(0x7f8d280b3fb0 msgr2=0x7f8d280b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.205+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 shutdown_connections 2024-04-15T15:15:51.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.206+0000 7f8d2ccba640 1 -- 172.21.15.70:0/2620640559 wait complete. 2024-04-15T15:15:51.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.323+0000 7f0f21ed9640 1 Processor -- start 2024-04-15T15:15:51.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.323+0000 7f0f21ed9640 1 -- start start 2024-04-15T15:15:51.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f21ed9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0f1c071d10 0x7f0f1c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f21ed9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f1c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f21ed9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c15db40 con 0x7f0f1c072d80 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f21ed9640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0f1c072170 con 0x7f0f1c071d10 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f21ed9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1c15cfc0 con 0x7f0f1c0d0bd0 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f1b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0f1c072d80 0x7f0f1c0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55894/0 (socket says 172.21.15.70:55894) 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f1b7fe640 1 -- 172.21.15.70:0/2079235537 learned_addr learned my addr 172.21.15.70:0/2079235537 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f1c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2079235537 >> v2:172.21.15.96:3300/0 conn(0x7f0f1c071d10 0x7f0f1c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2694868070 0 0) 0x7f0f1c15db40 con 0x7f0f1c072d80 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.324+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0f08003520 con 0x7f0f1c072d80 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2262815565 0 0) 0x7f0f08003520 con 0x7f0f1c072d80 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 msgr2=0x7f0f1c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f1c06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 >> v2:172.21.15.96:3300/0 conn(0x7f0f1c071d10 msgr2=0x7f0f1c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 --2- 172.21.15.70:0/2079235537 >> v2:172.21.15.96:3300/0 conn(0x7f0f1c071d10 0x7f0f1c0d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c15d2a0 con 0x7f0f1c072d80 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2079235537 >> v2:172.21.15.96:3300/0 conn(0x7f0f1c071d10 0x7f0f1c0d0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f0f0c001670 con 0x7f0f1c072d80 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 >> v1:172.21.15.70:6789/0 conn(0x7f0f1c072d80 legacy=0x7f0f1c0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f080039b0 0x7f0f08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c072170 con 0x7f0f08004d40 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c15cfc0 con 0x7f0f080039b0 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0f0800b9c0 con 0x7f0f1c0d0bd0 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1b7fe640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f080039b0 0x7f0f08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 -- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 msgr2=0x7f0f08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c0d0bd0 0x7f0f08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 -- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f080039b0 msgr2=0x7f0f08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f080039b0 0x7f0f08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.325+0000 7f0f1bfff640 1 -- 172.21.15.70:0/2079235537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f08003520 con 0x7f0f08004d40 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.326+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f080051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009fc0 tx=0x7f0f08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a53088e797e699 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.326+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14039070 con 0x7f0f08004d40 2024-04-15T15:15:51.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.326+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1402cc20 con 0x7f0f08004d40 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.326+0000 7f0f1affd640 1 -- 172.21.15.70:0/2079235537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14009c10 con 0x7f0f08004d40 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2079235537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 msgr2=0x7f0f080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2079235537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f080051c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f0f14009fc0 tx=0x7f0f08005830 comp rx=0 tx=0).stop 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f1bfff640 1 -- 172.21.15.70:0/2079235537 reap_dead start 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2079235537 shutdown_connections 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2079235537 >> 172.21.15.70:0/2079235537 conn(0x7f0f1c06be00 msgr2=0x7f0f1c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2079235537 shutdown_connections 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.327+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2079235537 wait complete. 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.328+0000 7f0f21ed9640 1 Processor -- start 2024-04-15T15:15:51.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.328+0000 7f0f21ed9640 1 -- start start 2024-04-15T15:15:51.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 0x7f0f1c0b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 0x7f0f1c152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c0afaf0 con 0x7f0f08004d40 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c15d860 con 0x7f0f1c0ccc80 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1c15cb90 con 0x7f0f1c152b30 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45540/0 (socket says 172.21.15.70:45540) 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 -- 172.21.15.70:0/2884150621 learned_addr learned my addr 172.21.15.70:0/2884150621 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 0x7f0f1c0b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 0x7f0f1c152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 -- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 msgr2=0x7f0f1c152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 0x7f0f1c152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 -- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 msgr2=0x7f0f1c0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 0x7f0f1c0b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 -- 172.21.15.70:0/2884150621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c152090 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 0x7f0f1c0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1bfff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 0x7f0f1c152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f1b7fe640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c007310 tx=0x7f0f0c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00f9c0 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f0c018070 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c00fd80 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1c0cd1b0 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.329+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1c152610 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.330+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1c152810 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.331+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0f0c01d460 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.331+0000 7f0f18ff9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 0x7f0efc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.331+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6735+0+0 (secure 0 0 0) 0x7f0f0c08bfe0 con 0x7f0f08004d40 2024-04-15T15:15:51.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.331+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 0x7f0efc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.332+0000 7f0f13fff640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 0x7f0efc06edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c15a730 tx=0x7f0f04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.335+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0f1c152810 con 0x7f0f08004d40 2024-04-15T15:15:51.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.519+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f1c0cfac0 con 0x7f0f08004d40 2024-04-15T15:15:51.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.520+0000 7f0f18ff9640 1 -- 172.21.15.70:0/2884150621 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f0f0c05a0a0 con 0x7f0f08004d40 2024-04-15T15:15:51.523 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 msgr2=0x7f0efc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 0x7f0efc06edb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c15a730 tx=0x7f0f04008040 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 msgr2=0x7f0f1c0cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c007310 tx=0x7f0f0c0077d0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 shutdown_connections 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0f1c152b30 0x7f0f1c152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0f1c0ccc80 0x7f0f1c0b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0efc06c8f0 0x7f0efc06edb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 --2- 172.21.15.70:0/2884150621 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0f08004d40 0x7f0f1c0cf520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 >> 172.21.15.70:0/2884150621 conn(0x7f0f1c06be00 msgr2=0x7f0f1c074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 shutdown_connections 2024-04-15T15:15:51.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.522+0000 7f0f21ed9640 1 -- 172.21.15.70:0/2884150621 wait complete. 2024-04-15T15:15:51.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.639+0000 7fc80aa35640 1 Processor -- start 2024-04-15T15:15:51.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.639+0000 7fc80aa35640 1 -- start start 2024-04-15T15:15:51.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc80aa35640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc8040d2360 0x7fc8040d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc80aa35640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8040b47a0 con 0x7fc80415eae0 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc80aa35640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8040b4a80 con 0x7fc80415e700 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc80aa35640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc8040b53b0 con 0x7fc8040d2360 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc808fd2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc80415eae0 0x7fc8040cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55904/0 (socket says 172.21.15.70:55904) 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.640+0000 7fc808fd2640 1 -- 172.21.15.70:0/2011072604 learned_addr learned my addr 172.21.15.70:0/2011072604 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc8037fe640 1 --2- 172.21.15.70:0/2011072604 >> v2:172.21.15.96:3300/0 conn(0x7fc8040d2360 0x7fc8040d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2904846718 0 0) 0x7fc8040b47a0 con 0x7fc80415eae0 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7ec003520 con 0x7fc80415eae0 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2415990809 0 0) 0x7fc7ec003520 con 0x7fc80415eae0 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc8037fe640 1 -- 172.21.15.70:0/2011072604 >> v1:172.21.15.96:6789/0 conn(0x7fc80415e700 legacy=0x7fc8040b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc8037fe640 1 -- 172.21.15.70:0/2011072604 >> v1:172.21.15.70:6789/0 conn(0x7fc80415eae0 legacy=0x7fc8040cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.641+0000 7fc8037fe640 1 -- 172.21.15.70:0/2011072604 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8040b4d60 con 0x7fc8040d2360 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc8037fe640 1 --2- 172.21.15.70:0/2011072604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8040d2360 0x7fc8040d2b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8040b5530 tx=0x7fc7f802b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5942b6a6f0f6771 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8039070 con 0x7fc8040d2360 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f80314f0 con 0x7fc8040d2360 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc802ffd640 1 -- 172.21.15.70:0/2011072604 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc0019f0 con 0x7fc8040d2360 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc80aa35640 1 -- 172.21.15.70:0/2011072604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8040d2360 msgr2=0x7fc8040d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.642+0000 7fc80aa35640 1 --2- 172.21.15.70:0/2011072604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8040d2360 0x7fc8040d2b60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8040b5530 tx=0x7fc7f802b810 comp rx=0 tx=0).stop 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.643+0000 7fc80aa35640 1 -- 172.21.15.70:0/2011072604 shutdown_connections 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.643+0000 7fc80aa35640 1 --2- 172.21.15.70:0/2011072604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc8040d2360 0x7fc8040d2b60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.643+0000 7fc80aa35640 1 -- 172.21.15.70:0/2011072604 >> 172.21.15.70:0/2011072604 conn(0x7fc80408bf60 msgr2=0x7fc8040d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.643+0000 7fc80aa35640 1 -- 172.21.15.70:0/2011072604 shutdown_connections 2024-04-15T15:15:51.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.643+0000 7fc80aa35640 1 -- 172.21.15.70:0/2011072604 wait complete. 2024-04-15T15:15:51.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.644+0000 7fc80aa35640 1 Processor -- start 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.644+0000 7fc80aa35640 1 -- start start 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 0x7fc80415ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 0x7fc804160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040d1d40 con 0x7fc8040d2360 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040b4450 con 0x7fc80415e700 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc8040d1ec0 con 0x7fc80415eae0 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45554/0 (socket says 172.21.15.70:45554) 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 -- 172.21.15.70:0/3734488495 learned_addr learned my addr 172.21.15.70:0/3734488495 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc8037fe640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 0x7fc80415ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 -- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 msgr2=0x7fc804160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc808fd2640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 0x7fc804160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 0x7fc804160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 -- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 msgr2=0x7fc80415ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 0x7fc80415ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 -- 172.21.15.70:0/3734488495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc804161240 con 0x7fc8040d2360 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc808fd2640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 0x7fc804160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc8037fe640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 0x7fc80415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc803fff640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400c450 tx=0x7fc7f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f40052d0 con 0x7fc8040d2360 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f4005490 con 0x7fc8040d2360 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc804161520 con 0x7fc8040d2360 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f4003880 con 0x7fc8040d2360 2024-04-15T15:15:51.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.645+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8040b80d0 con 0x7fc8040d2360 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.647+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8040c8be0 con 0x7fc8040d2360 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.647+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc7f4003a20 con 0x7fc8040d2360 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.647+0000 7fc800ff9640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 0x7fc7f006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.648+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc7f408bb00 con 0x7fc8040d2360 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.648+0000 7fc8037fe640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 0x7fc7f006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.648+0000 7fc8037fe640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 0x7fc7f006ede0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8036000 tx=0x7fc7f8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.652+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7f4059bc0 con 0x7fc8040d2360 2024-04-15T15:15:51.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.826+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc8040689e0 con 0x7fc8040d2360 2024-04-15T15:15:51.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.826+0000 7fc800ff9640 1 -- 172.21.15.70:0/3734488495 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc7f40594c0 con 0x7fc8040d2360 2024-04-15T15:15:51.830 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 msgr2=0x7fc7f006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 0x7fc7f006ede0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8036000 tx=0x7fc7f8008030 comp rx=0 tx=0).stop 2024-04-15T15:15:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 msgr2=0x7fc8040b7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc7f400c450 tx=0x7fc7f400c910 comp rx=0 tx=0).stop 2024-04-15T15:15:51.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 shutdown_connections 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc80415eae0 0x7fc804160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc80415e700 0x7fc80415ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc7f006c920 0x7fc7f006ede0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 --2- 172.21.15.70:0/3734488495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc8040d2360 0x7fc8040b7b90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 >> 172.21.15.70:0/3734488495 conn(0x7fc80408bf60 msgr2=0x7fc8040d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 shutdown_connections 2024-04-15T15:15:51.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.828+0000 7fc80aa35640 1 -- 172.21.15.70:0/3734488495 wait complete. 2024-04-15T15:15:51.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:51.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.946+0000 7f3b27055640 1 Processor -- start 2024-04-15T15:15:51.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.946+0000 7f3b27055640 1 -- start start 2024-04-15T15:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.946+0000 7f3b27055640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 0x7f3b200cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b27055640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b201606b0 0x7f3b200d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b27055640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b200cd6a0 con 0x7f3b2015cbc0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b27055640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3b200cd980 con 0x7f3b201606b0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b27055640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3b200cdc60 con 0x7f3b201602d0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b201606b0 0x7f3b200d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3b201606b0 0x7f3b200d0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45564/0 (socket says 172.21.15.70:45564) 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 -- 172.21.15.70:0/1931598353 learned_addr learned my addr 172.21.15.70:0/1931598353 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b255f2640 1 --1- 172.21.15.70:0/1931598353 >> v1:172.21.15.70:6789/0 conn(0x7f3b2015cbc0 0x7f3b2015d050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55916/0 (socket says 172.21.15.70:55916) 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b24df1640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 0x7f3b200cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 -- 172.21.15.70:0/1931598353 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 msgr2=0x7f3b200cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 0x7f3b200cd000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 -- 172.21.15.70:0/1931598353 >> v1:172.21.15.70:6789/0 conn(0x7f3b2015cbc0 legacy=0x7f3b2015d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b1ffff640 1 -- 172.21.15.70:0/1931598353 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b200cdf40 con 0x7f3b201606b0 2024-04-15T15:15:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.947+0000 7f3b24df1640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 0x7f3b200cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b1ffff640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b200d0ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009e50 tx=0x7f3b1002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da422478efcbe658 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b1f7fe640 1 -- 172.21.15.70:0/1931598353 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1002cb00 con 0x7f3b201606b0 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b1f7fe640 1 -- 172.21.15.70:0/1931598353 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b1002cca0 con 0x7f3b201606b0 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b1f7fe640 1 -- 172.21.15.70:0/1931598353 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b10034610 con 0x7f3b201606b0 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 -- 172.21.15.70:0/1931598353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 msgr2=0x7f3b200d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b200d0ef0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3b10009e50 tx=0x7f3b1002bf60 comp rx=0 tx=0).stop 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 -- 172.21.15.70:0/1931598353 shutdown_connections 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b200d0ef0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 --2- 172.21.15.70:0/1931598353 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b201602d0 0x7f3b200cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.948+0000 7f3b27055640 1 -- 172.21.15.70:0/1931598353 >> 172.21.15.70:0/1931598353 conn(0x7f3b200d2460 msgr2=0x7f3b200d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:51.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.949+0000 7f3b27055640 1 -- 172.21.15.70:0/1931598353 shutdown_connections 2024-04-15T15:15:51.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.949+0000 7f3b27055640 1 -- 172.21.15.70:0/1931598353 wait complete. 2024-04-15T15:15:51.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 Processor -- start 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 -- start start 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b2015cbc0 0x7f3b200b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 0x7f3b20157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.950+0000 7f3b27055640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b20160b10 con 0x7f3b201606b0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b27055640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b200d1430 con 0x7f3b201602d0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b27055640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3b2009bb50 con 0x7f3b2015cbc0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45568/0 (socket says 172.21.15.70:45568) 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 -- 172.21.15.70:0/625961461 learned_addr learned my addr 172.21.15.70:0/625961461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b24df1640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b2015cbc0 0x7f3b200b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b1ffff640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 0x7f3b20157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 -- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b2015cbc0 msgr2=0x7f3b200b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b2015cbc0 0x7f3b200b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 -- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 msgr2=0x7f3b20157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 0x7f3b20157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 -- 172.21.15.70:0/625961461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b200c5af0 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b1ffff640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 0x7f3b20157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b255f2640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800c430 tx=0x7f3b0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b080074e0 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b08007680 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b080049e0 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b200b5e10 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.951+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b2015e4e0 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.952+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b200689e0 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.953+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3b0801a020 con 0x7f3b201606b0 2024-04-15T15:15:51.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.953+0000 7f3b1d7fa640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 0x7f3b0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:51.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.953+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3b0808b760 con 0x7f3b201606b0 2024-04-15T15:15:51.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.953+0000 7f3b24df1640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 0x7f3b0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:51.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.954+0000 7f3b24df1640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 0x7f3b0006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800a990 tx=0x7f3b18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:51.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:51.958+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b08010300 con 0x7f3b201606b0 2024-04-15T15:15:52.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.132+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3b200b4e20 con 0x7f3b201606b0 2024-04-15T15:15:52.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.133+0000 7f3b1d7fa640 1 -- 172.21.15.70:0/625961461 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f3b08059790 con 0x7f3b201606b0 2024-04-15T15:15:52.136 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 msgr2=0x7f3b0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 0x7f3b0006edb0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800a990 tx=0x7f3b18008040 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 msgr2=0x7f3b201e59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3b0800c430 tx=0x7f3b0800c8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 shutdown_connections 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b0006c8f0 0x7f3b0006edb0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b201606b0 0x7f3b201e59a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b201602d0 0x7f3b20157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 --2- 172.21.15.70:0/625961461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b2015cbc0 0x7f3b200b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 >> 172.21.15.70:0/625961461 conn(0x7f3b200d2460 msgr2=0x7f3b2015a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 shutdown_connections 2024-04-15T15:15:52.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.135+0000 7f3b27055640 1 -- 172.21.15.70:0/625961461 wait complete. 2024-04-15T15:15:52.161 INFO:tasks.cephfs_test_runner:Starting test: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:15:52.161 INFO:tasks.cephfs_test_runner:test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:15:52.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.263+0000 7f13a67f2640 1 Processor -- start 2024-04-15T15:15:52.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.263+0000 7f13a67f2640 1 -- start start 2024-04-15T15:15:52.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a67f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009a9d0 0x7f13a0160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a67f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f13a00701f0 0x7f13a015fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a67f2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13a0070720 con 0x7f13a009adb0 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a67f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f13a009ba90 con 0x7f13a00701f0 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a67f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f13a0160070 con 0x7f13a009a9d0 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f13a00701f0 0x7f13a015fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f13a00701f0 0x7f13a015fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45570/0 (socket says 172.21.15.70:45570) 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 -- 172.21.15.70:0/3819784207 learned_addr learned my addr 172.21.15.70:0/3819784207 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f13a4d8f640 1 --1- 172.21.15.70:0/3819784207 >> v1:172.21.15.70:6789/0 conn(0x7f13a009adb0 0x7f13a00d0d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:55926/0 (socket says 172.21.15.70:55926) 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139ffff640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009a9d0 0x7f13a0160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 -- 172.21.15.70:0/3819784207 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009a9d0 msgr2=0x7f13a0160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009a9d0 0x7f13a0160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 -- 172.21.15.70:0/3819784207 >> v1:172.21.15.70:6789/0 conn(0x7f13a009adb0 legacy=0x7f13a00d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139f7fe640 1 -- 172.21.15.70:0/3819784207 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a0070a90 con 0x7f13a00701f0 2024-04-15T15:15:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.264+0000 7f139effd640 1 -- 172.21.15.70:0/3819784207 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2847651868 0 0) 0x7f13a0070720 con 0x7f13a009adb0 2024-04-15T15:15:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f139f7fe640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a015fb30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13940098a0 tx=0x7f139402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2f32d0cea5c15a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f139effd640 1 -- 172.21.15.70:0/3819784207 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394034070 con 0x7f13a00701f0 2024-04-15T15:15:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f139effd640 1 -- 172.21.15.70:0/3819784207 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1394030840 con 0x7f13a00701f0 2024-04-15T15:15:52.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f139effd640 1 -- 172.21.15.70:0/3819784207 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394030c10 con 0x7f13a00701f0 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f13a67f2640 1 -- 172.21.15.70:0/3819784207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 msgr2=0x7f13a015fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.265+0000 7f13a67f2640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a015fb30 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13940098a0 tx=0x7f139402b980 comp rx=0 tx=0).stop 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 -- 172.21.15.70:0/3819784207 shutdown_connections 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a015fb30 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 --2- 172.21.15.70:0/3819784207 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009a9d0 0x7f13a0160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 -- 172.21.15.70:0/3819784207 >> 172.21.15.70:0/3819784207 conn(0x7f13a00d2460 msgr2=0x7f13a00d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 -- 172.21.15.70:0/3819784207 shutdown_connections 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.266+0000 7f13a67f2640 1 -- 172.21.15.70:0/3819784207 wait complete. 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.267+0000 7f13a67f2640 1 Processor -- start 2024-04-15T15:15:52.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.267+0000 7f13a67f2640 1 -- start start 2024-04-15T15:15:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13a009a9d0 0x7f13a0161490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009adb0 0x7f13a01619d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f13a0071050 con 0x7f13a00701f0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f13a00d08c0 con 0x7f13a009a9d0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f13a00b1c60 con 0x7f13a009adb0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45574/0 (socket says 172.21.15.70:45574) 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 -- 172.21.15.70:0/2435409567 learned_addr learned my addr 172.21.15.70:0/2435409567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a4d8f640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009adb0 0x7f13a01619d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139f7fe640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13a009a9d0 0x7f13a0161490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 -- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009adb0 msgr2=0x7f13a01619d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009adb0 0x7f13a01619d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 -- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13a009a9d0 msgr2=0x7f13a0161490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13a009a9d0 0x7f13a0161490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 -- 172.21.15.70:0/2435409567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a00af790 con 0x7f13a00701f0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139ffff640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f139000d800 tx=0x7f139000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139000b6c0 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1390019070 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139000ba80 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a00cab10 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.268+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a00cb3b0 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.269+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13a00689e0 con 0x7f13a00701f0 2024-04-15T15:15:52.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.270+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f139000bc20 con 0x7f13a00701f0 2024-04-15T15:15:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.270+0000 7f139cff9640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 0x7f138c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.270+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f139008c440 con 0x7f13a00701f0 2024-04-15T15:15:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.270+0000 7f139f7fe640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 0x7f138c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.271+0000 7f139f7fe640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 0x7f138c06edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f139403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.274+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f139001c360 con 0x7f13a00701f0 2024-04-15T15:15:52.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.453+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f13a00ceab0 con 0x7f13a00701f0 2024-04-15T15:15:52.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.454+0000 7f139cff9640 1 -- 172.21.15.70:0/2435409567 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7f139005a470 con 0x7f13a00701f0 2024-04-15T15:15:52.456 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:52.456 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 msgr2=0x7f138c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 0x7f138c06edb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1394036000 tx=0x7f139403b040 comp rx=0 tx=0).stop 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 msgr2=0x7f13a00af250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f139000d800 tx=0x7f139000dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 shutdown_connections 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f13a009adb0 0x7f13a01619d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f13a009a9d0 0x7f13a0161490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f138c06c8f0 0x7f138c06edb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 --2- 172.21.15.70:0/2435409567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f13a00701f0 0x7f13a00af250 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 >> 172.21.15.70:0/2435409567 conn(0x7f13a00d2460 msgr2=0x7f13a015e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 shutdown_connections 2024-04-15T15:15:52.459 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.456+0000 7f13a67f2640 1 -- 172.21.15.70:0/2435409567 wait complete. 2024-04-15T15:15:52.469 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:52.469 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:52.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.574+0000 7f7bd8b32640 1 Processor -- start 2024-04-15T15:15:52.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.574+0000 7f7bd8b32640 1 -- start start 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd8b32640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7bd40d0690 0x7f7bd40d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd8b32640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 0x7f7bd415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd8b32640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bd40d0fb0 con 0x7f7bd415ef70 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd8b32640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd415cb50 con 0x7f7bd40d0690 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd8b32640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd415bfd0 con 0x7f7bd40b3160 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7bd40d0690 0x7f7bd40d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7bd40d0690 0x7f7bd40d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42938/0 (socket says 172.21.15.70:42938) 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 -- 172.21.15.70:0/3095542123 learned_addr learned my addr 172.21.15.70:0/3095542123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/3095542123 >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 0x7f7bd415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 -- 172.21.15.70:0/3095542123 >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 msgr2=0x7f7bd415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 --2- 172.21.15.70:0/3095542123 >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 0x7f7bd415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 -- 172.21.15.70:0/3095542123 >> v1:172.21.15.96:6789/0 conn(0x7f7bd415ef70 legacy=0x7f7bd415f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bd259c640 1 -- 172.21.15.70:0/3095542123 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd415c2b0 con 0x7f7bd40d0690 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.575+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/3095542123 >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 0x7f7bd415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.576+0000 7f7bd259c640 1 --2- 172.21.15.70:0/3095542123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40d0690 0x7f7bd40d0a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7bd406fec0 tx=0x7f7bc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ba0a738721ce6d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.576+0000 7f7bd1d9b640 1 -- 172.21.15.70:0/3095542123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4035020 con 0x7f7bd40d0690 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.576+0000 7f7bd1d9b640 1 -- 172.21.15.70:0/3095542123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc4031870 con 0x7f7bd40d0690 2024-04-15T15:15:52.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.576+0000 7f7bd1d9b640 1 -- 172.21.15.70:0/3095542123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4031c40 con 0x7f7bd40d0690 2024-04-15T15:15:52.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 -- 172.21.15.70:0/3095542123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40d0690 msgr2=0x7f7bd40d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/3095542123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40d0690 0x7f7bd40d0a70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7bd406fec0 tx=0x7f7bc402b910 comp rx=0 tx=0).stop 2024-04-15T15:15:52.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 -- 172.21.15.70:0/3095542123 shutdown_connections 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/3095542123 >> v2:172.21.15.96:3300/0 conn(0x7f7bd40b3160 0x7f7bd415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/3095542123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40d0690 0x7f7bd40d0a70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 -- 172.21.15.70:0/3095542123 >> 172.21.15.70:0/3095542123 conn(0x7f7bd406be00 msgr2=0x7f7bd406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 -- 172.21.15.70:0/3095542123 shutdown_connections 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.577+0000 7f7bd8b32640 1 -- 172.21.15.70:0/3095542123 wait complete. 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.578+0000 7f7bd8b32640 1 Processor -- start 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.578+0000 7f7bd8b32640 1 -- start start 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 0x7f7bd41519e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7bd415ef70 0x7f7bd4151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40b4fb0 con 0x7f7bd40b3160 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd415c870 con 0x7f7bd40d0690 2024-04-15T15:15:52.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd415bbd0 con 0x7f7bd415ef70 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42942/0 (socket says 172.21.15.70:42942) 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 -- 172.21.15.70:0/4093219012 learned_addr learned my addr 172.21.15.70:0/4093219012 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 0x7f7bd41519e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd2d9d640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7bd415ef70 0x7f7bd4151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 -- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7bd415ef70 msgr2=0x7f7bd4151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7bd415ef70 0x7f7bd4151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 -- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 msgr2=0x7f7bd41519e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 0x7f7bd41519e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 -- 172.21.15.70:0/4093219012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd4152830 con 0x7f7bd40b3160 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 0x7f7bd41519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd259c640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7bc402b950 tx=0x7f7bc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4040070 con 0x7f7bd40b3160 2024-04-15T15:15:52.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bc403a420 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bd414df80 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bc4035040 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.579+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd414e540 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.580+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd40689e0 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.581+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7bc4038020 con 0x7f7bd40b3160 2024-04-15T15:15:52.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.581+0000 7f7bcaffd640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 0x7f7bb406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.581+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7bc40addf0 con 0x7f7bd40b3160 2024-04-15T15:15:52.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.581+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 0x7f7bb406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.582+0000 7f7bcbfff640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 0x7f7bb406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40cd080 tx=0x7f7bbc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.586+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7bc407beb0 con 0x7f7bd40b3160 2024-04-15T15:15:52.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.760+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7bd414ec40 con 0x7f7bd40b3160 2024-04-15T15:15:52.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.760+0000 7f7bcaffd640 1 -- 172.21.15.70:0/4093219012 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7bc4082080 con 0x7f7bd40b3160 2024-04-15T15:15:52.762 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 msgr2=0x7f7bb406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 0x7f7bb406edb0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7bd40cd080 tx=0x7f7bbc008040 comp rx=0 tx=0).stop 2024-04-15T15:15:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 msgr2=0x7f7bd415e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7bc402b950 tx=0x7f7bc40046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 shutdown_connections 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7bd415ef70 0x7f7bd4151f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7bd40d0690 0x7f7bd41519e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7bb406c8f0 0x7f7bb406edb0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 --2- 172.21.15.70:0/4093219012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7bd40b3160 0x7f7bd415e8d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 >> 172.21.15.70:0/4093219012 conn(0x7f7bd406be00 msgr2=0x7f7bd40cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 shutdown_connections 2024-04-15T15:15:52.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.762+0000 7f7bd8b32640 1 -- 172.21.15.70:0/4093219012 wait complete. 2024-04-15T15:15:52.775 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.879+0000 7f379900d640 1 Processor -- start 2024-04-15T15:15:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.879+0000 7f379900d640 1 -- start start 2024-04-15T15:15:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379900d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f379415e790 0x7f37940b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379900d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37940d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379900d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f379415eb70 con 0x7f37940b7be0 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379900d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f37940d3790 con 0x7f379415e790 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379900d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37940d45b0 con 0x7f37940d1cf0 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f37940b7be0 0x7f37940d15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56974/0 (socket says 172.21.15.70:56974) 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379259c640 1 -- 172.21.15.70:0/2431898334 learned_addr learned my addr 172.21.15.70:0/2431898334 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f379359e640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37940d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f3792d9d640 1 --2- 172.21.15.70:0/2431898334 >> v2:172.21.15.96:3300/0 conn(0x7f379415e790 0x7f37940b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2343372753 0 0) 0x7f379415eb70 con 0x7f37940b7be0 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.880+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3780003520 con 0x7f37940b7be0 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2696604377 0 0) 0x7f3780003520 con 0x7f37940b7be0 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 msgr2=0x7f37940d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37940d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 >> v2:172.21.15.96:3300/0 conn(0x7f379415e790 msgr2=0x7f37940b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 --2- 172.21.15.70:0/2431898334 >> v2:172.21.15.96:3300/0 conn(0x7f379415e790 0x7f37940b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37940d3c80 con 0x7f37940b7be0 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f377c001690 con 0x7f37940b7be0 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 >> v1:172.21.15.70:6789/0 conn(0x7f37940b7be0 legacy=0x7f37940d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37800039d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3780004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3780004d40 0x7f37800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37940d3790 con 0x7f3780003f50 2024-04-15T15:15:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37940d45b0 con 0x7f3780004d40 2024-04-15T15:15:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f378000bac0 con 0x7f37940d1cf0 2024-04-15T15:15:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3780004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3792d9d640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37800039d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379359e640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3780004d40 0x7f37800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 -- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 msgr2=0x7f37800039d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37800039d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 -- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3780004d40 msgr2=0x7f37800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3780004d40 0x7f37800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379259c640 1 -- 172.21.15.70:0/2431898334 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3780003520 con 0x7f3780003f50 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f3792d9d640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37940d1cf0 0x7f37800039d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.881+0000 7f379359e640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3780004d40 0x7f37800051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379259c640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3780004800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f37940cb690 tx=0x7f3780005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39653fb6e6dfce12 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c007d60 con 0x7f3780003f50 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f377c030070 con 0x7f3780003f50 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f3791d9b640 1 -- 172.21.15.70:0/2431898334 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f377c005d10 con 0x7f3780003f50 2024-04-15T15:15:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379900d640 1 -- 172.21.15.70:0/2431898334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 msgr2=0x7f3780004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379900d640 1 --2- 172.21.15.70:0/2431898334 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3780004800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f37940cb690 tx=0x7f3780005830 comp rx=0 tx=0).stop 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379359e640 1 -- 172.21.15.70:0/2431898334 reap_dead start 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379900d640 1 -- 172.21.15.70:0/2431898334 shutdown_connections 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.882+0000 7f379900d640 1 -- 172.21.15.70:0/2431898334 >> 172.21.15.70:0/2431898334 conn(0x7f3794067b40 msgr2=0x7f3794067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.883+0000 7f379900d640 1 -- 172.21.15.70:0/2431898334 shutdown_connections 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.883+0000 7f379900d640 1 -- 172.21.15.70:0/2431898334 wait complete. 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.884+0000 7f379900d640 1 Processor -- start 2024-04-15T15:15:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.884+0000 7f379900d640 1 -- start start 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37941541b0 0x7f37940ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37940ca830 0x7f37941523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37940d4a40 con 0x7f3780003f50 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37940d4240 con 0x7f37940ca830 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379900d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37940d4bc0 con 0x7f37941541b0 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42962/0 (socket says 172.21.15.70:42962) 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 -- 172.21.15.70:0/3498594504 learned_addr learned my addr 172.21.15.70:0/3498594504 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379259c640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37941541b0 0x7f37940ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f379359e640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37940ca830 0x7f37941523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 -- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37941541b0 msgr2=0x7f37940ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37941541b0 0x7f37940ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 -- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37940ca830 msgr2=0x7f37941523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37940ca830 0x7f37941523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 -- 172.21.15.70:0/3498594504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3794153110 con 0x7f3780003f50 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f3792d9d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f378c0101c0 tx=0x7f378c010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c01a030 con 0x7f3780003f50 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.885+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f378c003990 con 0x7f3780003f50 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.886+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f378c003d60 con 0x7f3780003f50 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.886+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37941533f0 con 0x7f3780003f50 2024-04-15T15:15:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.886+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37941539b0 con 0x7f3780003f50 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.887+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37940689e0 con 0x7f3780003f50 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.887+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f378c01d070 con 0x7f3780003f50 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.887+0000 7f378b7fe640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 0x7f377406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.887+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f378c08cd40 con 0x7f3780003f50 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.887+0000 7f379259c640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 0x7f377406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.888+0000 7f379259c640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 0x7f377406ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f377c034000 tx=0x7f377c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:52.892+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f378c05a720 con 0x7f3780003f50 2024-04-15T15:15:53.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.065+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f37940d1830 con 0x7f3780003f50 2024-04-15T15:15:53.067 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.066+0000 7f378b7fe640 1 -- 172.21.15.70:0/3498594504 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f378c05ea20 con 0x7f3780003f50 2024-04-15T15:15:53.068 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:53.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 msgr2=0x7f377406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 0x7f377406ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f377c034000 tx=0x7f377c007000 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 msgr2=0x7f3794150ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f378c0101c0 tx=0x7f378c010680 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 shutdown_connections 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37940ca830 0x7f37941523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37941541b0 0x7f37940ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f377406c920 0x7f377406ede0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 --2- 172.21.15.70:0/3498594504 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3780003f50 0x7f3794150ae0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 >> 172.21.15.70:0/3498594504 conn(0x7f3794067b40 msgr2=0x7f3794161270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 shutdown_connections 2024-04-15T15:15:53.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.068+0000 7f379900d640 1 -- 172.21.15.70:0/3498594504 wait complete. 2024-04-15T15:15:53.080 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:53.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.184+0000 7f6168041640 1 Processor -- start 2024-04-15T15:15:53.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.184+0000 7f6168041640 1 -- start start 2024-04-15T15:15:53.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f6168041640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f616015ec10 0x7f61600b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f6168041640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6160067e30 con 0x7f61600b53c0 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f6168041640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61600b3c10 con 0x7f616015e830 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f6168041640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f61600b3ef0 con 0x7f616015ec10 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f616015ec10 0x7f61600b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f616015ec10 0x7f61600b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42966/0 (socket says 172.21.15.70:42966) 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61665de640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f61600b53c0 0x7f61600aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56976/0 (socket says 172.21.15.70:56976) 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 -- 172.21.15.70:0/2626504230 learned_addr learned my addr 172.21.15.70:0/2626504230 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:53.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 -- 172.21.15.70:0/2626504230 >> v1:172.21.15.96:6789/0 conn(0x7f616015e830 legacy=0x7f6160067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 -- 172.21.15.70:0/2626504230 >> v1:172.21.15.70:6789/0 conn(0x7f61600b53c0 legacy=0x7f61600aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f6164ddb640 1 -- 172.21.15.70:0/2626504230 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3223279840 0 0) 0x7f6160067e30 con 0x7f61600b53c0 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.185+0000 7f61655dc640 1 -- 172.21.15.70:0/2626504230 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61600b41d0 con 0x7f616015ec10 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f61655dc640 1 --2- 172.21.15.70:0/2626504230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015ec10 0x7f61600b4cb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6150009870 tx=0x7f615002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9eb99c92dcf0a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f6164ddb640 1 -- 172.21.15.70:0/2626504230 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150035020 con 0x7f616015ec10 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f6164ddb640 1 -- 172.21.15.70:0/2626504230 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6150031870 con 0x7f616015ec10 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f6164ddb640 1 -- 172.21.15.70:0/2626504230 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6150031c40 con 0x7f616015ec10 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f6168041640 1 -- 172.21.15.70:0/2626504230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015ec10 msgr2=0x7f61600b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.186+0000 7f6168041640 1 --2- 172.21.15.70:0/2626504230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015ec10 0x7f61600b4cb0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f6150009870 tx=0x7f615002b910 comp rx=0 tx=0).stop 2024-04-15T15:15:53.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.187+0000 7f6168041640 1 -- 172.21.15.70:0/2626504230 shutdown_connections 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.187+0000 7f6168041640 1 --2- 172.21.15.70:0/2626504230 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015ec10 0x7f61600b4cb0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.187+0000 7f6168041640 1 -- 172.21.15.70:0/2626504230 >> 172.21.15.70:0/2626504230 conn(0x7f6160073360 msgr2=0x7f616006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.187+0000 7f6168041640 1 -- 172.21.15.70:0/2626504230 shutdown_connections 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.187+0000 7f6168041640 1 -- 172.21.15.70:0/2626504230 wait complete. 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.188+0000 7f6168041640 1 Processor -- start 2024-04-15T15:15:53.192 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.188+0000 7f6168041640 1 -- start start 2024-04-15T15:15:53.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.188+0000 7f6168041640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 0x7f616013eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.188+0000 7f6168041640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6160156f30 0x7f6160158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f616015ff60 con 0x7f616015e830 2024-04-15T15:15:53.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61600b4790 con 0x7f61600b53c0 2024-04-15T15:15:53.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f61601600e0 con 0x7f6160156f30 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42974/0 (socket says 172.21.15.70:42974) 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 -- 172.21.15.70:0/1431363150 learned_addr learned my addr 172.21.15.70:0/1431363150 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 -- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6160156f30 msgr2=0x7f6160158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6160156f30 0x7f6160158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 -- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 msgr2=0x7f616013eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6165ddd640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 0x7f616013eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 0x7f616013eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 -- 172.21.15.70:0/1431363150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61601573a0 con 0x7f616015e830 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6165ddd640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 0x7f616013eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61655dc640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6150009870 tx=0x7f61500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f615003ad60 con 0x7f616015e830 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61500443f0 con 0x7f616015e830 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6160158e70 con 0x7f616015e830 2024-04-15T15:15:53.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61500447e0 con 0x7f616015e830 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.189+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f616013f4e0 con 0x7f616015e830 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.190+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61600689e0 con 0x7f616015e830 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.191+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f615004b060 con 0x7f616015e830 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.191+0000 7f61567fc640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 0x7f614406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.191+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f61500ae160 con 0x7f616015e830 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.191+0000 7f6165ddd640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 0x7f614406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.192+0000 7f6165ddd640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 0x7f614406edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f61601f7db0 tx=0x7f615c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.196+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f615003f070 con 0x7f616015e830 2024-04-15T15:15:53.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.371+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f61600afbf0 con 0x7f616015e830 2024-04-15T15:15:53.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.372+0000 7f61567fc640 1 -- 172.21.15.70:0/1431363150 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f615002f320 con 0x7f616015e830 2024-04-15T15:15:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.373+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 msgr2=0x7f614406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.373+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 0x7f614406edb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f61601f7db0 tx=0x7f615c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.373+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 msgr2=0x7f61601569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.373+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6150009870 tx=0x7f61500046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 shutdown_connections 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6160156f30 0x7f6160158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f614406c8f0 0x7f614406edb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f616015e830 0x7f61601569f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 --2- 172.21.15.70:0/1431363150 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61600b53c0 0x7f616013eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 >> 172.21.15.70:0/1431363150 conn(0x7f6160073360 msgr2=0x7f616006e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 shutdown_connections 2024-04-15T15:15:53.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.374+0000 7f6168041640 1 -- 172.21.15.70:0/1431363150 wait complete. 2024-04-15T15:15:53.387 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.491+0000 7f4d8fc8f640 1 Processor -- start 2024-04-15T15:15:53.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.491+0000 7f4d8fc8f640 1 -- start start 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4d8809a9d0 0x7f4d88160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4d880701f0 0x7f4d8815fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809adb0 0x7f4d880d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4d8809ba90 con 0x7f4d880701f0 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4d88160070 con 0x7f4d8809a9d0 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8fc8f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4d880d08c0 con 0x7f4d8809adb0 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8d22a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4d880701f0 0x7f4d8815fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8d22a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4d880701f0 0x7f4d8815fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:42990/0 (socket says 172.21.15.70:42990) 2024-04-15T15:15:53.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.492+0000 7f4d8d22a640 1 -- 172.21.15.70:0/1155742592 learned_addr learned my addr 172.21.15.70:0/1155742592 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:53.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8da2b640 1 --2- 172.21.15.70:0/1155742592 >> v2:172.21.15.96:3300/0 conn(0x7f4d8809a9d0 0x7f4d88160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8e22c640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809adb0 0x7f4d880d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 -- 172.21.15.70:0/1155742592 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809adb0 msgr2=0x7f4d880d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809adb0 0x7f4d880d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 -- 172.21.15.70:0/1155742592 >> v2:172.21.15.96:3300/0 conn(0x7f4d8809a9d0 msgr2=0x7f4d88160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/1155742592 >> v2:172.21.15.96:3300/0 conn(0x7f4d8809a9d0 0x7f4d88160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 -- 172.21.15.70:0/1155742592 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88070720 con 0x7f4d880701f0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d8815fb30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4d880d30a0 tx=0x7f4d7802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a990165bbf35f7c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.493+0000 7f4d8ca29640 1 -- 172.21.15.70:0/1155742592 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78035020 con 0x7f4d880701f0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8ca29640 1 -- 172.21.15.70:0/1155742592 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d78031640 con 0x7f4d880701f0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8ca29640 1 -- 172.21.15.70:0/1155742592 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7802d040 con 0x7f4d880701f0 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/1155742592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 msgr2=0x7f4d8815fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d8815fb30 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4d880d30a0 tx=0x7f4d7802b8f0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/1155742592 shutdown_connections 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809adb0 0x7f4d880d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/1155742592 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d8815fb30 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/1155742592 >> v2:172.21.15.96:3300/0 conn(0x7f4d8809a9d0 0x7f4d88160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/1155742592 >> 172.21.15.70:0/1155742592 conn(0x7f4d880d2460 msgr2=0x7f4d880d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/1155742592 shutdown_connections 2024-04-15T15:15:53.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.494+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/1155742592 wait complete. 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.495+0000 7f4d8fc8f640 1 Processor -- start 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.495+0000 7f4d8fc8f640 1 -- start start 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809a9d0 0x7f4d88150850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d8809adb0 0x7f4d88161010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88071050 con 0x7f4d880701f0 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d88070d70 con 0x7f4d8809adb0 2024-04-15T15:15:53.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8fc8f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4d880b1c60 con 0x7f4d8809a9d0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43004/0 (socket says 172.21.15.70:43004) 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 -- 172.21.15.70:0/624969911 learned_addr learned my addr 172.21.15.70:0/624969911 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8e22c640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d8809adb0 0x7f4d88161010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809a9d0 0x7f4d88150850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 -- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809a9d0 msgr2=0x7f4d88150850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809a9d0 0x7f4d88150850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 -- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d8809adb0 msgr2=0x7f4d88161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d8809adb0 0x7f4d88161010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.496+0000 7f4d8da2b640 1 -- 172.21.15.70:0/624969911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88161910 con 0x7f4d880701f0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d8da2b640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4d84007e90 tx=0x7f4d8400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84014620 con 0x7f4d880701f0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d84019070 con 0x7f4d880701f0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d84014a00 con 0x7f4d880701f0 2024-04-15T15:15:53.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d88161bf0 con 0x7f4d880701f0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.497+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d881621b0 con 0x7f4d880701f0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.498+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d880689e0 con 0x7f4d880701f0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.498+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4d84014ba0 con 0x7f4d880701f0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.499+0000 7f4d7e7fc640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.499+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.499+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4d8408c7f0 con 0x7f4d880701f0 2024-04-15T15:15:53.503 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.499+0000 7f4d8d22a640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d78036000 tx=0x7f4d78008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.503+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d84011260 con 0x7f4d880701f0 2024-04-15T15:15:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.677+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4d880d3370 con 0x7f4d880701f0 2024-04-15T15:15:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.677+0000 7f4d7e7fc640 1 -- 172.21.15.70:0/624969911 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f4d8405a820 con 0x7f4d880701f0 2024-04-15T15:15:53.679 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:53.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 msgr2=0x7f4d6c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d78036000 tx=0x7f4d78008030 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 msgr2=0x7f4d881629b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4d84007e90 tx=0x7f4d8400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 shutdown_connections 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d8809adb0 0x7f4d88161010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d8809a9d0 0x7f4d88150850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d6c06c8f0 0x7f4d6c06edb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 --2- 172.21.15.70:0/624969911 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d880701f0 0x7f4d881629b0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 >> 172.21.15.70:0/624969911 conn(0x7f4d880d2460 msgr2=0x7f4d880b2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.679+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 shutdown_connections 2024-04-15T15:15:53.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.680+0000 7f4d8fc8f640 1 -- 172.21.15.70:0/624969911 wait complete. 2024-04-15T15:15:53.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:15:53.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.793+0000 7f7f10932640 1 Processor -- start 2024-04-15T15:15:53.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.793+0000 7f7f10932640 1 -- start start 2024-04-15T15:15:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f10932640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f10932640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7f0c15fb30 0x7f7f0c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f10932640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f0c070720 con 0x7f7f0c0701f0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f10932640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7f0c09ba90 con 0x7f7f0c15fb30 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f10932640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c0d08c0 con 0x7f7f0c09a9d0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f09d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7f0c0701f0 0x7f7f0c09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56978/0 (socket says 172.21.15.70:56978) 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.794+0000 7f7f09d9b640 1 -- 172.21.15.70:0/1390004923 learned_addr learned my addr 172.21.15.70:0/1390004923 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0a59c640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0ad9d640 1 --2- 172.21.15.70:0/1390004923 >> v2:172.21.15.96:3300/0 conn(0x7f7f0c15fb30 0x7f7f0c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 314461489 0 0) 0x7f7f0c070720 con 0x7f7f0c0701f0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7ef4003520 con 0x7f7f0c0701f0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2722115909 0 0) 0x7f7ef4003520 con 0x7f7f0c0701f0 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0ad9d640 1 -- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 msgr2=0x7f7f0c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0ad9d640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0ad9d640 1 -- 172.21.15.70:0/1390004923 >> v1:172.21.15.70:6789/0 conn(0x7f7f0c0701f0 legacy=0x7f7f0c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0ad9d640 1 -- 172.21.15.70:0/1390004923 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c070a90 con 0x7f7f0c15fb30 2024-04-15T15:15:53.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.795+0000 7f7f0a59c640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:53.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f0ad9d640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c0d0d50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009dc0 tx=0x7f7f0402bd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50e4eea1084ab71d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04039070 con 0x7f7f0c15fb30 2024-04-15T15:15:53.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f0402c880 con 0x7f7f0c15fb30 2024-04-15T15:15:53.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f0959a640 1 -- 172.21.15.70:0/1390004923 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f00001670 con 0x7f7f0c15fb30 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f10932640 1 -- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 msgr2=0x7f7f0c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f10932640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c0d0d50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009dc0 tx=0x7f7f0402bd50 comp rx=0 tx=0).stop 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f10932640 1 -- 172.21.15.70:0/1390004923 shutdown_connections 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f10932640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c0d0d50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.796+0000 7f7f10932640 1 --2- 172.21.15.70:0/1390004923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.797+0000 7f7f10932640 1 -- 172.21.15.70:0/1390004923 >> 172.21.15.70:0/1390004923 conn(0x7f7f0c0d2460 msgr2=0x7f7f0c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.797+0000 7f7f10932640 1 -- 172.21.15.70:0/1390004923 shutdown_connections 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.797+0000 7f7f10932640 1 -- 172.21.15.70:0/1390004923 wait complete. 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.798+0000 7f7f10932640 1 Processor -- start 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.798+0000 7f7f10932640 1 -- start start 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c1549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c15ffc0 con 0x7f7f0c0701f0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c071050 con 0x7f7f0c15fb30 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7f0c0b1c60 con 0x7f7f0c09a9d0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43018/0 (socket says 172.21.15.70:43018) 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 -- 172.21.15.70:0/2973059702 learned_addr learned my addr 172.21.15.70:0/2973059702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0ad9d640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c1549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f09d9b640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 -- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 msgr2=0x7f7f0c15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 -- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 msgr2=0x7f7f0c1549b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c1549b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 -- 172.21.15.70:0/2973059702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c15bd70 con 0x7f7f0c0701f0 2024-04-15T15:15:53.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f09d9b640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f0a59c640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00ec10 tx=0x7f7efc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc017070 con 0x7f7f0c0701f0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7efc00cc30 con 0x7f7f0c0701f0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7efc013470 con 0x7f7f0c0701f0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c15db60 con 0x7f7f0c0701f0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.799+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c15c2f0 con 0x7f7f0c0701f0 2024-04-15T15:15:53.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.800+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c0689e0 con 0x7f7f0c0701f0 2024-04-15T15:15:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.801+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7efc01a070 con 0x7f7f0c0701f0 2024-04-15T15:15:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.801+0000 7f7efaffd640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 0x7f7eec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.801+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7efc08bec0 con 0x7f7f0c0701f0 2024-04-15T15:15:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.801+0000 7f7f09d9b640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 0x7f7eec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:53.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.802+0000 7f7f09d9b640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 0x7f7eec06edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b1670 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:53.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:53.805+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7efc059ef0 con 0x7f7f0c0701f0 2024-04-15T15:15:54.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.049+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7f7f0c0d3370 con 0x7f7f0c0701f0 2024-04-15T15:15:54.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.422+0000 7f7efaffd640 1 -- 172.21.15.70:0/2973059702 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v68) ==== 140+0+0 (secure 0 0 0) 0x7f7efc0597f0 con 0x7f7f0c0701f0 2024-04-15T15:15:54.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 msgr2=0x7f7eec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 0x7f7eec06edb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c0b1670 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 msgr2=0x7f7f0c0b50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7efc00ec10 tx=0x7f7efc00c520 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 shutdown_connections 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7f0c15fb30 0x7f7f0c1549b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7f0c09a9d0 0x7f7f0c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7eec06c8f0 0x7f7eec06edb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 --2- 172.21.15.70:0/2973059702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7f0c0701f0 0x7f7f0c0b50c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 >> 172.21.15.70:0/2973059702 conn(0x7f7f0c0d2460 msgr2=0x7f7f0c0b2a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 shutdown_connections 2024-04-15T15:15:54.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.425+0000 7f7f10932640 1 -- 172.21.15.70:0/2973059702 wait complete. 2024-04-15T15:15:54.438 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-15T15:15:54.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.544+0000 7f7762529640 1 Processor -- start 2024-04-15T15:15:54.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.544+0000 7f7762529640 1 -- start start 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f7762529640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f775c0cff10 0x7f775c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f7762529640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775c0b1f40 con 0x7f775c0b1480 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f7762529640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f775c0cda30 con 0x7f775c15fda0 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f7762529640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f775c0cdd10 con 0x7f775c0cff10 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f775c0b1480 0x7f775c0cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56980/0 (socket says 172.21.15.70:56980) 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775b7fe640 1 -- 172.21.15.70:0/1671775305 learned_addr learned my addr 172.21.15.70:0/1671775305 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f7760ac6640 1 --2- 172.21.15.70:0/1671775305 >> v2:172.21.15.96:3300/0 conn(0x7f775c0cff10 0x7f775c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704462341 0 0) 0x7f775c0b1f40 con 0x7f775c0b1480 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7744003520 con 0x7f775c0b1480 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3474648622 0 0) 0x7f7744003520 con 0x7f775c0b1480 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 >> v2:172.21.15.96:3300/0 conn(0x7f775c0cff10 msgr2=0x7f775c0b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 --2- 172.21.15.70:0/1671775305 >> v2:172.21.15.96:3300/0 conn(0x7f775c0cff10 0x7f775c0b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 >> v1:172.21.15.96:6789/0 conn(0x7f775c15fda0 legacy=0x7f775c0ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c0cdff0 con 0x7f775c0b1480 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f7750001670 con 0x7f775c0b1480 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 >> v1:172.21.15.70:6789/0 conn(0x7f775c0b1480 legacy=0x7f775c0cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440039b0 0x7f7744003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7744004800 0x7f7744004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.545+0000 7f775affd640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 0x7f774400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f775c0cdd10 con 0x7f77440051c0 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f774400c060 con 0x7f7744004800 2024-04-15T15:15:54.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f774400c260 con 0x7f77440039b0 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 0x7f774400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f775b7fe640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7744004800 0x7f7744004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f775bfff640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440039b0 0x7f7744003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 -- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440039b0 msgr2=0x7f7744003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440039b0 0x7f7744003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 -- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7744004800 msgr2=0x7f7744004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7744004800 0x7f7744004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 -- 172.21.15.70:0/1671775305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7744003520 con 0x7f77440051c0 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.546+0000 7f7760ac6640 1 -- 172.21.15.70:0/1671775305 reap_dead start 2024-04-15T15:15:54.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7760ac6640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 0x7f774400b9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7754009fc0 tx=0x7f7744005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8cfb9cb18530fe server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754009c10 con 0x7f77440051c0 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775402cca0 con 0x7f77440051c0 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f775affd640 1 -- 172.21.15.70:0/1671775305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754003bd0 con 0x7f77440051c0 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 -- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 msgr2=0x7f774400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 0x7f774400b9c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7754009fc0 tx=0x7f7744005830 comp rx=0 tx=0).stop 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 -- 172.21.15.70:0/1671775305 shutdown_connections 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 --2- 172.21.15.70:0/1671775305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77440051c0 0x7f774400b9c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 -- 172.21.15.70:0/1671775305 >> 172.21.15.70:0/1671775305 conn(0x7f775c0b6870 msgr2=0x7f775c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 -- 172.21.15.70:0/1671775305 shutdown_connections 2024-04-15T15:15:54.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.547+0000 7f7762529640 1 -- 172.21.15.70:0/1671775305 wait complete. 2024-04-15T15:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.548+0000 7f7762529640 1 Processor -- start 2024-04-15T15:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 -- start start 2024-04-15T15:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440051c0 0x7f775c06f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f775c09dcc0 0x7f775c09e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f775c0b0790 con 0x7f775c1508b0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f775c160180 con 0x7f775c09dcc0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7762529640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f775c0b18a0 con 0x7f77440051c0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.549+0000 7f7760ac6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43034/0 (socket says 172.21.15.70:43034) 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 -- 172.21.15.70:0/3517889850 learned_addr learned my addr 172.21.15.70:0/3517889850 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f775bfff640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440051c0 0x7f775c06f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f775b7fe640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f775c09dcc0 0x7f775c09e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 -- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440051c0 msgr2=0x7f775c06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440051c0 0x7f775c06f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 -- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f775c09dcc0 msgr2=0x7f775c09e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f775c09dcc0 0x7f775c09e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 -- 172.21.15.70:0/3517889850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f775c1f2510 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7760ac6640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f775402c6d0 tx=0x7f7754009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754042020 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f775402ce50 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f775c1f27f0 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754009c10 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.550+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f775c1f2200 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.551+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f775c0689e0 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.552+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7754045020 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.552+0000 7f7758ff9640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 0x7f774806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.552+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f77540ae6d0 con 0x7f775c1508b0 2024-04-15T15:15:54.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.552+0000 7f775bfff640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 0x7f774806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:54.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.553+0000 7f775bfff640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 0x7f774806edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f774c007fd0 tx=0x7f774c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:54.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.556+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f775404c020 con 0x7f775c1508b0 2024-04-15T15:15:54.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.731+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f775c070860 con 0x7f775c1508b0 2024-04-15T15:15:54.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.733+0000 7f7758ff9640 1 -- 172.21.15.70:0/3517889850 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f775407c140 con 0x7f775c1508b0 2024-04-15T15:15:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 msgr2=0x7f774806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 0x7f774806edb0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f774c007fd0 tx=0x7f774c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:54.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 msgr2=0x7f775c150d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f775402c6d0 tx=0x7f7754009750 comp rx=0 tx=0).stop 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 shutdown_connections 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f774806c8f0 0x7f774806edb0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f775c1508b0 0x7f775c150d30 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f775c09dcc0 0x7f775c09e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 --2- 172.21.15.70:0/3517889850 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f77440051c0 0x7f775c06f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:54.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 >> 172.21.15.70:0/3517889850 conn(0x7f775c0b6870 msgr2=0x7f775c0b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:54.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 shutdown_connections 2024-04-15T15:15:54.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:54.736+0000 7f7762529640 1 -- 172.21.15.70:0/3517889850 wait complete. 2024-04-15T15:15:54.749 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:15:54.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:15:54.786 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:15:54.836 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:15:54.836 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:15:54.836 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:54.852 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:15:54.852 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:54.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:55.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.050+0000 7fa837ec5640 1 Processor -- start 2024-04-15T15:15:55.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.051+0000 7fa837ec5640 1 -- start start 2024-04-15T15:15:55.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.051+0000 7fa837ec5640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa83015e700 0x7fa8300b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.051+0000 7fa837ec5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8300b47a0 con 0x7fa8300d2360 2024-04-15T15:15:55.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.051+0000 7fa837ec5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8300b4a80 con 0x7fa8300d3270 2024-04-15T15:15:55.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.051+0000 7fa837ec5640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa8300b53b0 con 0x7fa83015e700 2024-04-15T15:15:55.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa835460640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa8300d2360 0x7fa8300d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:56994/0 (socket says 172.21.15.70:56994) 2024-04-15T15:15:55.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa835460640 1 -- 172.21.15.70:0/2657122501 learned_addr learned my addr 172.21.15.70:0/2657122501 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:55.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa835c61640 1 --2- 172.21.15.70:0/2657122501 >> v2:172.21.15.96:3300/0 conn(0x7fa83015e700 0x7fa8300b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33872036 0 0) 0x7fa8300b47a0 con 0x7fa8300d2360 2024-04-15T15:15:55.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa81c003520 con 0x7fa8300d2360 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2848956516 0 0) 0x7fa81c003520 con 0x7fa8300d2360 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 >> v2:172.21.15.96:3300/0 conn(0x7fa83015e700 msgr2=0x7fa8300b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 --2- 172.21.15.70:0/2657122501 >> v2:172.21.15.96:3300/0 conn(0x7fa83015e700 0x7fa8300b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 >> v1:172.21.15.96:6789/0 conn(0x7fa8300d3270 legacy=0x7fa8300cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8300b4d60 con 0x7fa8300d2360 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fa82c001670 con 0x7fa8300d2360 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 >> v1:172.21.15.70:6789/0 conn(0x7fa8300d2360 legacy=0x7fa8300d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa81c0039b0 0x7fa81c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 0x7fa81c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c0051c0 0x7fa81c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300b53b0 con 0x7fa81c004800 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa81c00c090 con 0x7fa81c0051c0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa81c00c290 con 0x7fa81c0039b0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa835460640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 0x7fa81c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa835c61640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa81c0039b0 0x7fa81c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.052+0000 7fa836462640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c0051c0 0x7fa81c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 -- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa81c0039b0 msgr2=0x7fa81c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa81c0039b0 0x7fa81c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 -- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c0051c0 msgr2=0x7fa81c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c0051c0 0x7fa81c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa836462640 1 -- 172.21.15.70:0/2657122501 reap_dead start 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 -- 172.21.15.70:0/2657122501 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81c003520 con 0x7fa81c004800 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa835460640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 0x7fa81c004c80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c240 tx=0x7fa81c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20415943c391fc65 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c007cf0 con 0x7fa81c004800 2024-04-15T15:15:55.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.053+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c030070 con 0x7fa81c004800 2024-04-15T15:15:55.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa834c5f640 1 -- 172.21.15.70:0/2657122501 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c005d10 con 0x7fa81c004800 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 -- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 msgr2=0x7fa81c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 0x7fa81c004c80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c240 tx=0x7fa81c005830 comp rx=0 tx=0).stop 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 -- 172.21.15.70:0/2657122501 shutdown_connections 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2657122501 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa81c004800 0x7fa81c004c80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 -- 172.21.15.70:0/2657122501 >> 172.21.15.70:0/2657122501 conn(0x7fa83008bf60 msgr2=0x7fa8300d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 -- 172.21.15.70:0/2657122501 shutdown_connections 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.054+0000 7fa837ec5640 1 -- 172.21.15.70:0/2657122501 wait complete. 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.055+0000 7fa837ec5640 1 Processor -- start 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.055+0000 7fa837ec5640 1 -- start start 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c004800 0x7fa830152ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8300cd990 0x7fa8300cde10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300d1d40 con 0x7fa83006db60 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa8300b4450 con 0x7fa81c004800 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa837ec5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa8300d1ec0 con 0x7fa8300cd990 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43050/0 (socket says 172.21.15.70:43050) 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 -- 172.21.15.70:0/2031832140 learned_addr learned my addr 172.21.15.70:0/2031832140 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835c61640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c004800 0x7fa830152ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa836462640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8300cd990 0x7fa8300cde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 -- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8300cd990 msgr2=0x7fa8300cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8300cd990 0x7fa8300cde10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 -- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c004800 msgr2=0x7fa830152ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c004800 0x7fa830152ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.056+0000 7fa835460640 1 -- 172.21.15.70:0/2031832140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8300cc7d0 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa835460640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa82c02c540 tx=0x7fa8300b4970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c046070 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa82c004bf0 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa830153420 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c001c30 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.057+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8300ccd50 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.058+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8300d1070 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.058+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa82c007070 con 0x7fa83006db60 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.059+0000 7fa8227fc640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 0x7fa81406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.059+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa82c0ad180 con 0x7fa83006db60 2024-04-15T15:15:55.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.059+0000 7fa835c61640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 0x7fa81406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.059+0000 7fa835c61640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 0x7fa81406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa824009c30 tx=0x7fa824009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.063+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8300d1070 con 0x7fa83006db60 2024-04-15T15:15:55.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.240+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa8300d3690 con 0x7fa83006db60 2024-04-15T15:15:55.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.241+0000 7fa8227fc640 1 -- 172.21.15.70:0/2031832140 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fa8300d3690 con 0x7fa83006db60 2024-04-15T15:15:55.244 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:55.244 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 msgr2=0x7fa81406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 0x7fa81406edb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa824009c30 tx=0x7fa824009110 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 msgr2=0x7fa83006dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa82c02c540 tx=0x7fa8300b4970 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 shutdown_connections 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8300cd990 0x7fa8300cde10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa81406c8f0 0x7fa81406edb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa83006db60 0x7fa83006dfe0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 --2- 172.21.15.70:0/2031832140 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa81c004800 0x7fa830152ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.243+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 >> 172.21.15.70:0/2031832140 conn(0x7fa83008bf60 msgr2=0x7fa83015ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.244+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 shutdown_connections 2024-04-15T15:15:55.246 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.244+0000 7fa837ec5640 1 -- 172.21.15.70:0/2031832140 wait complete. 2024-04-15T15:15:55.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:55.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:55.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.362+0000 7f877a582640 1 Processor -- start 2024-04-15T15:15:55.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.362+0000 7f877a582640 1 -- start start 2024-04-15T15:15:55.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f877a582640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87740b5230 0x7f8774072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f877a582640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f877a582640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87740b4190 con 0x7f8774158770 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f877a582640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87740730d0 con 0x7f87740b5230 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f877a582640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f87740b4da0 con 0x7f8774072c60 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f876bfff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87740b5230 0x7f8774072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f8778b1f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f8773fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f8774158770 0x7f877415bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:60734/0 (socket says 172.21.15.70:60734) 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f8778b1f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:52860/0 (socket says 172.21.15.70:52860) 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f876bfff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f87740b5230 0x7f8774072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:58174/0 (socket says 172.21.15.70:58174) 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.363+0000 7f8773fff640 1 -- 172.21.15.70:0/4253800206 learned_addr learned my addr 172.21.15.70:0/4253800206 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f876bfff640 1 -- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 msgr2=0x7f877406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f876bfff640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f876bfff640 1 -- 172.21.15.70:0/4253800206 >> v1:172.21.15.96:6789/0 conn(0x7f8774158770 legacy=0x7f877415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f876bfff640 1 -- 172.21.15.70:0/4253800206 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87740b4470 con 0x7f87740b5230 2024-04-15T15:15:55.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f87737fe640 1 -- 172.21.15.70:0/4253800206 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470276835 0 0) 0x7f87740b4190 con 0x7f8774158770 2024-04-15T15:15:55.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.364+0000 7f8778b1f640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f876bfff640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740b5230 0x7f8774072720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7cf1a30bc47c0f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f87737fe640 1 -- 172.21.15.70:0/4253800206 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c035020 con 0x7f87740b5230 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f87737fe640 1 -- 172.21.15.70:0/4253800206 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c031830 con 0x7f87740b5230 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f87737fe640 1 -- 172.21.15.70:0/4253800206 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c031c00 con 0x7f87740b5230 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 -- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740b5230 msgr2=0x7f8774072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740b5230 0x7f8774072720 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 -- 172.21.15.70:0/4253800206 shutdown_connections 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774072c60 0x7f877406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 --2- 172.21.15.70:0/4253800206 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f87740b5230 0x7f8774072720 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 -- 172.21.15.70:0/4253800206 >> 172.21.15.70:0/4253800206 conn(0x7f877406be00 msgr2=0x7f877406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 -- 172.21.15.70:0/4253800206 shutdown_connections 2024-04-15T15:15:55.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.365+0000 7f877a582640 1 -- 172.21.15.70:0/4253800206 wait complete. 2024-04-15T15:15:55.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 Processor -- start 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 -- start start 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8774072c60 0x7f8774148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774158770 0x7f87740b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8774071fa0 con 0x7f87740b5230 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87740b4a30 con 0x7f8774072c60 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f877a582640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8774158b50 con 0x7f8774158770 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f876bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f876bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43056/0 (socket says 172.21.15.70:43056) 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.367+0000 7f876bfff640 1 -- 172.21.15.70:0/2401409771 learned_addr learned my addr 172.21.15.70:0/2401409771 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f8773fff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8774072c60 0x7f8774148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f8778b1f640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774158770 0x7f87740b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 -- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774158770 msgr2=0x7f87740b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774158770 0x7f87740b1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 -- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8774072c60 msgr2=0x7f8774148240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8774072c60 0x7f8774148240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 -- 172.21.15.70:0/2401409771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877415cd40 con 0x7f87740b5230 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f876bfff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c035070 con 0x7f87740b5230 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f875c004230 con 0x7f87740b5230 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877415d020 con 0x7f87740b5230 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f875c03a630 con 0x7f87740b5230 2024-04-15T15:15:55.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.368+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87740cc730 con 0x7f87740b5230 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.369+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877415d540 con 0x7f87740b5230 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.370+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f875c038020 con 0x7f87740b5230 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.370+0000 7f87717fa640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 0x7f875406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.370+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f875c0adb50 con 0x7f87740b5230 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.370+0000 7f8773fff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 0x7f875406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.370+0000 7f8773fff640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 0x7f875406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8764000f60 tx=0x7f8764009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.374+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f877415d540 con 0x7f87740b5230 2024-04-15T15:15:55.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.552+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f87740689e0 con 0x7f87740b5230 2024-04-15T15:15:55.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.552+0000 7f87717fa640 1 -- 172.21.15.70:0/2401409771 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f875c040070 con 0x7f87740b5230 2024-04-15T15:15:55.555 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 msgr2=0x7f875406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 0x7f875406edb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f8764000f60 tx=0x7f8764009040 comp rx=0 tx=0).stop 2024-04-15T15:15:55.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 msgr2=0x7f8774149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f875c0097a0 tx=0x7f875c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 shutdown_connections 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8774158770 0x7f87740b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f875406c8f0 0x7f875406edb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87740b5230 0x7f8774149730 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 --2- 172.21.15.70:0/2401409771 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8774072c60 0x7f8774148240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 >> 172.21.15.70:0/2401409771 conn(0x7f877406be00 msgr2=0x7f877406dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 shutdown_connections 2024-04-15T15:15:55.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.554+0000 7f877a582640 1 -- 172.21.15.70:0/2401409771 wait complete. 2024-04-15T15:15:55.568 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:55.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.672+0000 7fb5f364a640 1 Processor -- start 2024-04-15T15:15:55.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 -- start start 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb5ec15c8d0 0x7fb5ec0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5ec0d0350 con 0x7fb5ec09b650 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ec0b5b00 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.673+0000 7fb5f364a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb5ec0ce100 con 0x7fb5ec15c8d0 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43062/0 (socket says 172.21.15.70:43062) 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 -- 172.21.15.70:0/3555499758 learned_addr learned my addr 172.21.15.70:0/3555499758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f13e6640 1 --2- 172.21.15.70:0/3555499758 >> v2:172.21.15.96:3300/0 conn(0x7fb5ec15c8d0 0x7fb5ec0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 -- 172.21.15.70:0/3555499758 >> v2:172.21.15.96:3300/0 conn(0x7fb5ec15c8d0 msgr2=0x7fb5ec0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 --2- 172.21.15.70:0/3555499758 >> v2:172.21.15.96:3300/0 conn(0x7fb5ec15c8d0 0x7fb5ec0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 -- 172.21.15.70:0/3555499758 >> v1:172.21.15.96:6789/0 conn(0x7fb5ec09b650 legacy=0x7fb5ec09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.674+0000 7fb5f1be7640 1 -- 172.21.15.70:0/3555499758 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec0b54b0 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f1be7640 1 --2- 172.21.15.70:0/3555499758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec09b620 tx=0x7fb5e402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e887e602c2150f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5e3fff640 1 -- 172.21.15.70:0/3555499758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e402c9f0 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5e3fff640 1 -- 172.21.15.70:0/3555499758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e402cb90 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5e3fff640 1 -- 172.21.15.70:0/3555499758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e40343f0 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 -- 172.21.15.70:0/3555499758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec09b0e0 msgr2=0x7fb5ec0b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 --2- 172.21.15.70:0/3555499758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec09b620 tx=0x7fb5e402be50 comp rx=0 tx=0).stop 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 -- 172.21.15.70:0/3555499758 shutdown_connections 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 --2- 172.21.15.70:0/3555499758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec09b0e0 0x7fb5ec0b5f90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 --2- 172.21.15.70:0/3555499758 >> v2:172.21.15.96:3300/0 conn(0x7fb5ec15c8d0 0x7fb5ec0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.675+0000 7fb5f364a640 1 -- 172.21.15.70:0/3555499758 >> 172.21.15.70:0/3555499758 conn(0x7fb5ec0b3fb0 msgr2=0x7fb5ec0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.676+0000 7fb5f364a640 1 -- 172.21.15.70:0/3555499758 shutdown_connections 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.676+0000 7fb5f364a640 1 -- 172.21.15.70:0/3555499758 wait complete. 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 Processor -- start 2024-04-15T15:15:55.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 -- start start 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb5ec09b0e0 0x7fb5ec14f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5ec09b650 0x7fb5ec158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec0b6650 con 0x7fb5ec0cad30 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb5ec0b64d0 con 0x7fb5ec09b650 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f364a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb5ec0b67d0 con 0x7fb5ec09b0e0 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f1be7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f1be7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43072/0 (socket says 172.21.15.70:43072) 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.677+0000 7fb5f1be7640 1 -- 172.21.15.70:0/1876627630 learned_addr learned my addr 172.21.15.70:0/1876627630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 -- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb5ec09b0e0 msgr2=0x7fb5ec14f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb5ec09b0e0 0x7fb5ec14f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 -- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5ec09b650 msgr2=0x7fb5ec158a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5ec09b650 0x7fb5ec158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 -- 172.21.15.70:0/1876627630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5ec14f7f0 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f1be7640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4036040 tx=0x7fb5e4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e4043020 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5e403e450 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5ec157590 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5e403e860 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.678+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5ec1f6330 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.679+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5ec0d2190 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.680+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb5e4046020 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.680+0000 7fb5e1ffb640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 0x7fb5d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.680+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb5e40ae810 con 0x7fb5ec0cad30 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.680+0000 7fb5f13e6640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 0x7fb5d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.681+0000 7fb5f13e6640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 0x7fb5d006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec155860 tx=0x7fb5e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.685+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb5ec0d2190 con 0x7fb5ec0cad30 2024-04-15T15:15:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.859+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb5ec0690f0 con 0x7fb5ec0cad30 2024-04-15T15:15:55.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.860+0000 7fb5e1ffb640 1 -- 172.21.15.70:0/1876627630 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fb5e404d030 con 0x7fb5ec0cad30 2024-04-15T15:15:55.862 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 msgr2=0x7fb5d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 0x7fb5d006edb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec155860 tx=0x7fb5e8008040 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 msgr2=0x7fb5ec0cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fb5e4036040 tx=0x7fb5e4004a40 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 shutdown_connections 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb5d006c8f0 0x7fb5d006edb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5ec0cad30 0x7fb5ec0cb1c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb5ec09b650 0x7fb5ec158a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 --2- 172.21.15.70:0/1876627630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb5ec09b0e0 0x7fb5ec14f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 >> 172.21.15.70:0/1876627630 conn(0x7fb5ec0b3fb0 msgr2=0x7fb5ec071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 shutdown_connections 2024-04-15T15:15:55.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.862+0000 7fb5f364a640 1 -- 172.21.15.70:0/1876627630 wait complete. 2024-04-15T15:15:55.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:55.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 Processor -- start 2024-04-15T15:15:55.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 -- start start 2024-04-15T15:15:55.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 0x7fb8b00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b00d2360 0x7fb8b00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8b00b47a0 con 0x7fb8b015eae0 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb8b00b53b0 con 0x7fb8b00d2360 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.980+0000 7fb8b5a1b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b00b4a80 con 0x7fb8b015e700 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b00d2360 0x7fb8b00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8af7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb8b015eae0 0x7fb8b00cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57000/0 (socket says 172.21.15.70:57000) 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb8b00d2360 0x7fb8b00d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43082/0 (socket says 172.21.15.70:43082) 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8af7fe640 1 -- 172.21.15.70:0/3236739398 learned_addr learned my addr 172.21.15.70:0/3236739398 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 0x7fb8b00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 -- 172.21.15.70:0/3236739398 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 msgr2=0x7fb8b00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 0x7fb8b00b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 -- 172.21.15.70:0/3236739398 >> v1:172.21.15.70:6789/0 conn(0x7fb8b015eae0 legacy=0x7fb8b00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8ae7fc640 1 -- 172.21.15.70:0/3236739398 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639020410 0 0) 0x7fb8b00b47a0 con 0x7fb8b015eae0 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8a67fc640 1 -- 172.21.15.70:0/3236739398 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b00b4d60 con 0x7fb8b00d2360 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.981+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 0x7fb8b00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:55.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8a67fc640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b00d2360 0x7fb8b00d2b60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8caa99d9f64b302 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8ae7fc640 1 -- 172.21.15.70:0/3236739398 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898035020 con 0x7fb8b00d2360 2024-04-15T15:15:55.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8ae7fc640 1 -- 172.21.15.70:0/3236739398 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb898031830 con 0x7fb8b00d2360 2024-04-15T15:15:55.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8ae7fc640 1 -- 172.21.15.70:0/3236739398 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb898031c00 con 0x7fb8b00d2360 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3236739398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b00d2360 msgr2=0x7fb8b00d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b00d2360 0x7fb8b00d2b60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb8980097a0 tx=0x7fb89802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3236739398 shutdown_connections 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b00d2360 0x7fb8b00d2b60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3236739398 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b015e700 0x7fb8b00b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3236739398 >> 172.21.15.70:0/3236739398 conn(0x7fb8b008bf60 msgr2=0x7fb8b00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3236739398 shutdown_connections 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.982+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3236739398 wait complete. 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 Processor -- start 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 -- start start 2024-04-15T15:15:55.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 0x7fb8b0152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 0x7fb8b00ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b00b5d80 con 0x7fb8b015eae0 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b00b4450 con 0x7fb8b015e700 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.984+0000 7fb8b5a1b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb8b00d1d40 con 0x7fb8b00d2360 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43090/0 (socket says 172.21.15.70:43090) 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 -- 172.21.15.70:0/3430525099 learned_addr learned my addr 172.21.15.70:0/3430525099 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 0x7fb8b0152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8a67fc640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 0x7fb8b00ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 -- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 msgr2=0x7fb8b0152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 0x7fb8b0152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 -- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 msgr2=0x7fb8b00ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 0x7fb8b00ccac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:55.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 -- 172.21.15.70:0/3430525099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b01f7d60 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 0x7fb8b0152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8a67fc640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 0x7fb8b00ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8af7fe640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c730 tx=0x7fb8a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8018070 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8a800ce40 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b01f8040 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a800a860 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.985+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b01f8600 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.986+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b00689e0 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.987+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb8a801b070 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.987+0000 7fb8acff9640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 0x7fb89006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.987+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb8a808b520 con 0x7fb8b015eae0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.987+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 0x7fb89006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:55.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.988+0000 7fb8aeffd640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 0x7fb89006edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009960 tx=0x7fb8a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:55.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:55.991+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb8a8059550 con 0x7fb8b015eae0 2024-04-15T15:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.169+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb8b00ba240 con 0x7fb8b015eae0 2024-04-15T15:15:56.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.169+0000 7fb8acff9640 1 -- 172.21.15.70:0/3430525099 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb8a8058e50 con 0x7fb8b015eae0 2024-04-15T15:15:56.171 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 msgr2=0x7fb89006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 0x7fb89006edb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009960 tx=0x7fb8a0008040 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 msgr2=0x7fb8b015c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c730 tx=0x7fb8a800cbf0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 shutdown_connections 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb89006c8f0 0x7fb89006edb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b015eae0 0x7fb8b015c4f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb8b015e700 0x7fb8b00ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 --2- 172.21.15.70:0/3430525099 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb8b00d2360 0x7fb8b0152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 >> 172.21.15.70:0/3430525099 conn(0x7fb8b008bf60 msgr2=0x7fb8b006e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 shutdown_connections 2024-04-15T15:15:56.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.171+0000 7fb8b5a1b640 1 -- 172.21.15.70:0/3430525099 wait complete. 2024-04-15T15:15:56.184 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:56.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.288+0000 7fa353559640 1 Processor -- start 2024-04-15T15:15:56.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.288+0000 7fa353559640 1 -- start start 2024-04-15T15:15:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa353559640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa34c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.293 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa353559640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa34c0b47a0 con 0x7fa34c0d3270 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa353559640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa34c0b4a80 con 0x7fa34c0d2360 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa353559640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa34c0b53b0 con 0x7fa34c15e700 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa351af6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa34c0d3270 0x7fa34c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57008/0 (socket says 172.21.15.70:57008) 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa351af6640 1 -- 172.21.15.70:0/839667706 learned_addr learned my addr 172.21.15.70:0/839667706 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa3512f5640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa34c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2459768214 0 0) 0x7fa34c0b47a0 con 0x7fa34c0d3270 2024-04-15T15:15:56.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.289+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa338003520 con 0x7fa34c0d3270 2024-04-15T15:15:56.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3057500664 0 0) 0x7fa338003520 con 0x7fa34c0d3270 2024-04-15T15:15:56.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 msgr2=0x7fa34c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa34c0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 >> v1:172.21.15.96:6789/0 conn(0x7fa34c0d2360 legacy=0x7fa34c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c0b4d60 con 0x7fa34c0d3270 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fa344001670 con 0x7fa34c0d3270 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 >> v1:172.21.15.70:6789/0 conn(0x7fa34c0d3270 legacy=0x7fa34c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3380039e0 0x7fa338003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa338004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa3380051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c0b53b0 con 0x7fa338004d40 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa33800b9c0 con 0x7fa3380039e0 2024-04-15T15:15:56.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa33800bbc0 con 0x7fa34c15e700 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa3380051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa3512f5640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3380039e0 0x7fa338003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa350af4640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa338004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 -- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 msgr2=0x7fa338004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa338004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 -- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3380039e0 msgr2=0x7fa338003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa3380039e0 0x7fa338003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa351af6640 1 -- 172.21.15.70:0/839667706 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa338003520 con 0x7fa338004d40 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.290+0000 7fa350af4640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c15e700 0x7fa338004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.291+0000 7fa351af6640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa3380051c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c240 tx=0x7fa338005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc057147c2a71a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.291+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa344007d10 con 0x7fa338004d40 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.291+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa344030070 con 0x7fa338004d40 2024-04-15T15:15:56.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.291+0000 7fa343fff640 1 -- 172.21.15.70:0/839667706 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa344005d30 con 0x7fa338004d40 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 -- 172.21.15.70:0/839667706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 msgr2=0x7fa3380051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 --2- 172.21.15.70:0/839667706 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa3380051c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa34400c240 tx=0x7fa338005830 comp rx=0 tx=0).stop 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa351af6640 1 -- 172.21.15.70:0/839667706 reap_dead start 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 -- 172.21.15.70:0/839667706 shutdown_connections 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 -- 172.21.15.70:0/839667706 >> 172.21.15.70:0/839667706 conn(0x7fa34c08bf60 msgr2=0x7fa34c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 -- 172.21.15.70:0/839667706 shutdown_connections 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.292+0000 7fa353559640 1 -- 172.21.15.70:0/839667706 wait complete. 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.293+0000 7fa353559640 1 Processor -- start 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.293+0000 7fa353559640 1 -- start start 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 0x7fa34c0b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 0x7fa34c0b9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c0d1d40 con 0x7fa338004d40 2024-04-15T15:15:56.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa34c0b4450 con 0x7fa34c0b9670 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa34c0d1ec0 con 0x7fa34c0b35f0 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43120/0 (socket says 172.21.15.70:43120) 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 -- 172.21.15.70:0/3759960026 learned_addr learned my addr 172.21.15.70:0/3759960026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa351af6640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 0x7fa34c0b9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa350af4640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 0x7fa34c0b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 -- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 msgr2=0x7fa34c0b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 0x7fa34c0b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 -- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 msgr2=0x7fa34c0b9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 0x7fa34c0b9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 -- 172.21.15.70:0/3759960026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa34c0ba030 con 0x7fa338004d40 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa350af4640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 0x7fa34c0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa351af6640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 0x7fa34c0b9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa3512f5640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa348011e60 tx=0x7fa34800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348015630 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa3480157d0 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa34c0b7390 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.294+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa348015ba0 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.295+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa34c0b7950 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.295+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa34c0689e0 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.296+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa34801d400 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.296+0000 7fa341ffb640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 0x7fa33006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.296+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa34808d100 con 0x7fa338004d40 2024-04-15T15:15:56.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.296+0000 7fa350af4640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 0x7fa33006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.297+0000 7fa350af4640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 0x7fa33006edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa34c1f7ba0 tx=0x7fa33c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.301+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa34c0ba030 con 0x7fa338004d40 2024-04-15T15:15:56.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.478+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa34c0cc6a0 con 0x7fa338004d40 2024-04-15T15:15:56.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.478+0000 7fa341ffb640 1 -- 172.21.15.70:0/3759960026 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fa348019030 con 0x7fa338004d40 2024-04-15T15:15:56.480 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 msgr2=0x7fa33006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 0x7fa33006edb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa34c1f7ba0 tx=0x7fa33c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 msgr2=0x7fa34c1545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa348011e60 tx=0x7fa34800f580 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 shutdown_connections 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa34c0b9670 0x7fa34c0b9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa34c0b35f0 0x7fa34c0b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa33006c8f0 0x7fa33006edb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.480+0000 7fa353559640 1 --2- 172.21.15.70:0/3759960026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa338004d40 0x7fa34c1545d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.481+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 >> 172.21.15.70:0/3759960026 conn(0x7fa34c08bf60 msgr2=0x7fa34c06e170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.481+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 shutdown_connections 2024-04-15T15:15:56.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.481+0000 7fa353559640 1 -- 172.21.15.70:0/3759960026 wait complete. 2024-04-15T15:15:56.494 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.598+0000 7f3621edf640 1 Processor -- start 2024-04-15T15:15:56.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.598+0000 7f3621edf640 1 -- start start 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f3621edf640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 0x7f361c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f3621edf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f361c15fb30 0x7f361c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f3621edf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f361c070720 con 0x7f361c09a9d0 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f3621edf640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f361c09ba90 con 0x7f361c15fb30 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f3621edf640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f361c0d08c0 con 0x7f361c0701f0 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f361c15fb30 0x7f361c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f361c15fb30 0x7f361c0d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43122/0 (socket says 172.21.15.70:43122) 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 -- 172.21.15.70:0/1141105124 learned_addr learned my addr 172.21.15.70:0/1141105124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361affd640 1 --2- 172.21.15.70:0/1141105124 >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 0x7f361c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 -- 172.21.15.70:0/1141105124 >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 msgr2=0x7f361c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 --2- 172.21.15.70:0/1141105124 >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 0x7f361c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.599+0000 7f361bfff640 1 -- 172.21.15.70:0/1141105124 >> v1:172.21.15.96:6789/0 conn(0x7f361c09a9d0 legacy=0x7f361c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361bfff640 1 -- 172.21.15.70:0/1141105124 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c070a90 con 0x7f361c15fb30 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361affd640 1 --2- 172.21.15.70:0/1141105124 >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 0x7f361c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361bfff640 1 --2- 172.21.15.70:0/1141105124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c15fb30 0x7f361c0d0d50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f361c072180 tx=0x7f361402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3367eee20b496b5b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361a7fc640 1 -- 172.21.15.70:0/1141105124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614034070 con 0x7f361c15fb30 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361a7fc640 1 -- 172.21.15.70:0/1141105124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3614030860 con 0x7f361c15fb30 2024-04-15T15:15:56.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.600+0000 7f361a7fc640 1 -- 172.21.15.70:0/1141105124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614030c30 con 0x7f361c15fb30 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 -- 172.21.15.70:0/1141105124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c15fb30 msgr2=0x7f361c0d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 --2- 172.21.15.70:0/1141105124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c15fb30 0x7f361c0d0d50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f361c072180 tx=0x7f361402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 -- 172.21.15.70:0/1141105124 shutdown_connections 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 --2- 172.21.15.70:0/1141105124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c15fb30 0x7f361c0d0d50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 --2- 172.21.15.70:0/1141105124 >> v2:172.21.15.96:3300/0 conn(0x7f361c0701f0 0x7f361c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 -- 172.21.15.70:0/1141105124 >> 172.21.15.70:0/1141105124 conn(0x7f361c0d2460 msgr2=0x7f361c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 -- 172.21.15.70:0/1141105124 shutdown_connections 2024-04-15T15:15:56.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.601+0000 7f3621edf640 1 -- 172.21.15.70:0/1141105124 wait complete. 2024-04-15T15:15:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.602+0000 7f3621edf640 1 Processor -- start 2024-04-15T15:15:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.602+0000 7f3621edf640 1 -- start start 2024-04-15T15:15:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 0x7f361c09f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f361c15fb30 0x7f361c155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f361c15ffc0 con 0x7f361c09a9d0 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f361c071050 con 0x7f361c0701f0 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f3621edf640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f361c0b1c60 con 0x7f361c15fb30 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43128/0 (socket says 172.21.15.70:43128) 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 -- 172.21.15.70:0/3274642778 learned_addr learned my addr 172.21.15.70:0/3274642778 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361b7fe640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 0x7f361c09f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361bfff640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f361c15fb30 0x7f361c155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 -- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f361c15fb30 msgr2=0x7f361c155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f361c15fb30 0x7f361c155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 -- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 msgr2=0x7f361c09f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 0x7f361c09f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361affd640 1 -- 172.21.15.70:0/3274642778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361c09fac0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.603+0000 7f361b7fe640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 0x7f361c09f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f361affd640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f361000d800 tx=0x7f361000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3610004030 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36100041d0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f361c09fda0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36100096a0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.604+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f361c09f6d0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.605+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f361c0689e0 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.605+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3610009840 con 0x7f361c09a9d0 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.606+0000 7f3603fff640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 0x7f360806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.606+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f361008c2e0 con 0x7f361c09a9d0 2024-04-15T15:15:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.606+0000 7f361b7fe640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 0x7f360806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.606+0000 7f361b7fe640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 0x7f360806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f360c001e00 tx=0x7f360c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.610+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f361005a280 con 0x7f361c09a9d0 2024-04-15T15:15:56.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.796+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f361c06caf0 con 0x7f361c09a9d0 2024-04-15T15:15:56.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.796+0000 7f3603fff640 1 -- 172.21.15.70:0/3274642778 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f3610059b80 con 0x7f361c09a9d0 2024-04-15T15:15:56.798 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 msgr2=0x7f360806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 0x7f360806edb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f360c001e00 tx=0x7f360c009040 comp rx=0 tx=0).stop 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 msgr2=0x7f361c0b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f361000d800 tx=0x7f361000dcc0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 shutdown_connections 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f361c15fb30 0x7f361c155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f360806c8f0 0x7f360806edb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f361c09a9d0 0x7f361c0b68e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 --2- 172.21.15.70:0/3274642778 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f361c0701f0 0x7f361c09f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 >> 172.21.15.70:0/3274642778 conn(0x7f361c0d2460 msgr2=0x7f361c0b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 shutdown_connections 2024-04-15T15:15:56.801 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.798+0000 7f3621edf640 1 -- 172.21.15.70:0/3274642778 wait complete. 2024-04-15T15:15:56.811 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:15:56.811 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:56.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.982+0000 7f62c21bd640 1 Processor -- start 2024-04-15T15:15:56.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.982+0000 7f62c21bd640 1 -- start start 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62c21bd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 0x7f62bc06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62c21bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62c21bd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62bc06bf20 con 0x7f62bc15d8e0 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62c21bd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f62bc15cc80 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62c21bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc06c1c0 con 0x7f62bc15d460 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62baffd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62bb7fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 0x7f62bc06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62bbfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f62bc15d8e0 0x7f62bc0cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:60786/0 (socket says 172.21.15.70:60786) 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62baffd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:52934/0 (socket says 172.21.15.70:52934) 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.983+0000 7f62bbfff640 1 -- 172.21.15.70:0/335910230 learned_addr learned my addr 172.21.15.70:0/335910230 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:56.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.984+0000 7f62baffd640 1 -- 172.21.15.70:0/335910230 >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 msgr2=0x7f62bc06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.984+0000 7f62baffd640 1 --2- 172.21.15.70:0/335910230 >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 0x7f62bc06b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.984+0000 7f62baffd640 1 -- 172.21.15.70:0/335910230 >> v1:172.21.15.96:6789/0 conn(0x7f62bc15d8e0 legacy=0x7f62bc0cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.984+0000 7f62baffd640 1 -- 172.21.15.70:0/335910230 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc06c4a0 con 0x7f62bc15d460 2024-04-15T15:15:56.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.984+0000 7f62bb7fe640 1 --2- 172.21.15.70:0/335910230 >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 0x7f62bc06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62baffd640 1 --2- 172.21.15.70:0/335910230 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0097a0 tx=0x7f62ac02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c70cf5a52e19c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62ba7fc640 1 -- 172.21.15.70:0/335910230 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac0315a0 con 0x7f62bc15d460 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62ba7fc640 1 -- 172.21.15.70:0/335910230 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ac037070 con 0x7f62bc15d460 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62ba7fc640 1 -- 172.21.15.70:0/335910230 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac031960 con 0x7f62bc15d460 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 -- 172.21.15.70:0/335910230 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 msgr2=0x7f62bc0b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 --2- 172.21.15.70:0/335910230 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0097a0 tx=0x7f62ac02b840 comp rx=0 tx=0).stop 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 -- 172.21.15.70:0/335910230 shutdown_connections 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 --2- 172.21.15.70:0/335910230 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc0b2230 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 --2- 172.21.15.70:0/335910230 >> v2:172.21.15.96:3300/0 conn(0x7f62bc0ccfb0 0x7f62bc06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 -- 172.21.15.70:0/335910230 >> 172.21.15.70:0/335910230 conn(0x7f62bc09a9d0 msgr2=0x7f62bc0b3130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 -- 172.21.15.70:0/335910230 shutdown_connections 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.985+0000 7f62c21bd640 1 -- 172.21.15.70:0/335910230 wait complete. 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 Processor -- start 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 -- start start 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc14cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 0x7f62bc0b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc0cdaa0 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f62bc15cf40 con 0x7f62bc15d8e0 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.987+0000 7f62c21bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f62bc0cdc20 con 0x7f62bc15d460 2024-04-15T15:15:56.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43142/0 (socket says 172.21.15.70:43142) 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 -- 172.21.15.70:0/3843418291 learned_addr learned my addr 172.21.15.70:0/3843418291 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 -- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 msgr2=0x7f62bc14cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bbfff640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 0x7f62bc0b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62baffd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc14cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc14cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 -- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 msgr2=0x7f62bc0b2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 0x7f62bc0b2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 -- 172.21.15.70:0/3843418291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62bc0b34f0 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bbfff640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 0x7f62bc0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62baffd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc14cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62bb7fe640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e960 tx=0x7f62a400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4017070 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62a400ca00 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a400cdd0 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62bc0cf260 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.988+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62bc0cf820 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.989+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f62bc0690f0 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.990+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f62a401a070 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.990+0000 7f62b3fff640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 0x7f629c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.990+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f62a408bd70 con 0x7f62bc0ccfb0 2024-04-15T15:15:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.990+0000 7f62baffd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 0x7f629c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:56.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.991+0000 7f62baffd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 0x7f629c06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f62ac036000 tx=0x7f62ac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:56.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:56.995+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f62a4059e30 con 0x7f62bc0ccfb0 2024-04-15T15:15:57.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.178+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62bc06e850 con 0x7f62bc0ccfb0 2024-04-15T15:15:57.180 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.179+0000 7f62b3fff640 1 -- 172.21.15.70:0/3843418291 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7f62a4059730 con 0x7f62bc0ccfb0 2024-04-15T15:15:57.181 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:57.181 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 msgr2=0x7f629c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 0x7f629c06edb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f62ac036000 tx=0x7f62ac008030 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 msgr2=0x7f62bc0b3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f62a400e960 tx=0x7f62a400ee20 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 shutdown_connections 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f62bc15d8e0 0x7f62bc0b2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f62bc15d460 0x7f62bc14cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f629c06c8f0 0x7f629c06edb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 --2- 172.21.15.70:0/3843418291 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f62bc0ccfb0 0x7f62bc0b3940 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 >> 172.21.15.70:0/3843418291 conn(0x7f62bc09a9d0 msgr2=0x7f62bc0cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 shutdown_connections 2024-04-15T15:15:57.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.181+0000 7f62c21bd640 1 -- 172.21.15.70:0/3843418291 wait complete. 2024-04-15T15:15:57.194 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:57.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.300+0000 7faa8bc1b640 1 Processor -- start 2024-04-15T15:15:57.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 -- start start 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faa8415e700 0x7faa840b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7faa840d3270 0x7faa840cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa840b47a0 con 0x7faa840d2360 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7faa840b53b0 con 0x7faa8415e700 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.301+0000 7faa8bc1b640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7faa840b4a80 con 0x7faa840d3270 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faa8415e700 0x7faa840b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faa8415e700 0x7faa840b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43156/0 (socket says 172.21.15.70:43156) 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa891b6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7faa840d2360 0x7faa840d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57022/0 (socket says 172.21.15.70:57022) 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 -- 172.21.15.70:0/61038228 learned_addr learned my addr 172.21.15.70:0/61038228 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa8a1b8640 1 --2- 172.21.15.70:0/61038228 >> v2:172.21.15.96:3300/0 conn(0x7faa840d3270 0x7faa840cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 -- 172.21.15.70:0/61038228 >> v2:172.21.15.96:3300/0 conn(0x7faa840d3270 msgr2=0x7faa840cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 --2- 172.21.15.70:0/61038228 >> v2:172.21.15.96:3300/0 conn(0x7faa840d3270 0x7faa840cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 -- 172.21.15.70:0/61038228 >> v1:172.21.15.70:6789/0 conn(0x7faa840d2360 legacy=0x7faa840d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa899b7640 1 -- 172.21.15.70:0/61038228 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840b4d60 con 0x7faa8415e700 2024-04-15T15:15:57.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.302+0000 7faa889b5640 1 -- 172.21.15.70:0/61038228 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2765139544 0 0) 0x7faa840b47a0 con 0x7faa840d2360 2024-04-15T15:15:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa899b7640 1 --2- 172.21.15.70:0/61038228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa8415e700 0x7faa840b5840 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa840d1b00 tx=0x7faa8002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5086b27bb4462365 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa889b5640 1 -- 172.21.15.70:0/61038228 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80004230 con 0x7faa8415e700 2024-04-15T15:15:57.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa889b5640 1 -- 172.21.15.70:0/61038228 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa800043d0 con 0x7faa8415e700 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa889b5640 1 -- 172.21.15.70:0/61038228 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa80030840 con 0x7faa8415e700 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 -- 172.21.15.70:0/61038228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa8415e700 msgr2=0x7faa840b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/61038228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa8415e700 0x7faa840b5840 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faa840d1b00 tx=0x7faa8002b980 comp rx=0 tx=0).stop 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 -- 172.21.15.70:0/61038228 shutdown_connections 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/61038228 >> v2:172.21.15.96:3300/0 conn(0x7faa840d3270 0x7faa840cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/61038228 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa8415e700 0x7faa840b5840 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.303+0000 7faa8bc1b640 1 -- 172.21.15.70:0/61038228 >> 172.21.15.70:0/61038228 conn(0x7faa8408bf60 msgr2=0x7faa840d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.304+0000 7faa8bc1b640 1 -- 172.21.15.70:0/61038228 shutdown_connections 2024-04-15T15:15:57.307 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.304+0000 7faa8bc1b640 1 -- 172.21.15.70:0/61038228 wait complete. 2024-04-15T15:15:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 Processor -- start 2024-04-15T15:15:57.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 -- start start 2024-04-15T15:15:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa840d2360 0x7faa840b9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.310 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faa8415e700 0x7faa84158ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa840b5d80 con 0x7faa840d3270 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faa840b4450 con 0x7faa840d2360 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.305+0000 7faa8bc1b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faa840d1d40 con 0x7faa8415e700 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43158/0 (socket says 172.21.15.70:43158) 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 -- 172.21.15.70:0/2386465753 learned_addr learned my addr 172.21.15.70:0/2386465753 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 -- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faa8415e700 msgr2=0x7faa84158ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faa8415e700 0x7faa84158ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 -- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa840d2360 msgr2=0x7faa840b9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa840d2360 0x7faa840b9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 -- 172.21.15.70:0/2386465753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa840b82e0 con 0x7faa840d3270 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa891b6640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c450 tx=0x7faa7400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74018070 con 0x7faa840d3270 2024-04-15T15:15:57.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa74007500 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa840b85c0 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa74004a20 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.306+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa840b8b80 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.307+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa840689e0 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.308+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7faa7401b070 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.308+0000 7faa7a7fc640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 0x7faa6806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.308+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7faa7408b200 con 0x7faa840d3270 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.308+0000 7faa899b7640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 0x7faa6806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.309+0000 7faa899b7640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 0x7faa6806edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faa80035000 tx=0x7faa80030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.313+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faa74059230 con 0x7faa840d3270 2024-04-15T15:15:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.489+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa840ba500 con 0x7faa840d3270 2024-04-15T15:15:57.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.489+0000 7faa7a7fc640 1 -- 172.21.15.70:0/2386465753 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7faa74058b30 con 0x7faa840d3270 2024-04-15T15:15:57.491 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 msgr2=0x7faa6806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 0x7faa6806edb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faa80035000 tx=0x7faa80030000 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 msgr2=0x7faa840b7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7faa7400c450 tx=0x7faa7400c910 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 shutdown_connections 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faa8415e700 0x7faa84158ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7faa6806c8f0 0x7faa6806edb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa840d3270 0x7faa840b7b60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 --2- 172.21.15.70:0/2386465753 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faa840d2360 0x7faa840b9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 >> 172.21.15.70:0/2386465753 conn(0x7faa8408bf60 msgr2=0x7faa840cd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 shutdown_connections 2024-04-15T15:15:57.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.491+0000 7faa8bc1b640 1 -- 172.21.15.70:0/2386465753 wait complete. 2024-04-15T15:15:57.505 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.613+0000 7f4c03d7c640 1 Processor -- start 2024-04-15T15:15:57.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.613+0000 7f4c03d7c640 1 -- start start 2024-04-15T15:15:57.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c03d7c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c03d7c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 0x7f4bfc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c03d7c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bfc0d0fb0 con 0x7f4bfc0b3160 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c03d7c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc15bda0 con 0x7f4bfc0d0690 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c03d7c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4bfc15cb50 con 0x7f4bfc0ae280 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c01b18640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c01b18640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43172/0 (socket says 172.21.15.70:43172) 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.614+0000 7f4c01b18640 1 -- 172.21.15.70:0/2230301360 learned_addr learned my addr 172.21.15.70:0/2230301360 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c02319640 1 --2- 172.21.15.70:0/2230301360 >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 0x7f4bfc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c01b18640 1 -- 172.21.15.70:0/2230301360 >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 msgr2=0x7f4bfc0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c01b18640 1 --2- 172.21.15.70:0/2230301360 >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 0x7f4bfc0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c01b18640 1 -- 172.21.15.70:0/2230301360 >> v1:172.21.15.96:6789/0 conn(0x7f4bfc0b3160 legacy=0x7f4bfc15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c01b18640 1 -- 172.21.15.70:0/2230301360 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc15bfd0 con 0x7f4bfc0d0690 2024-04-15T15:15:57.617 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c02319640 1 --2- 172.21.15.70:0/2230301360 >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 0x7f4bfc0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c01b18640 1 --2- 172.21.15.70:0/2230301360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc06ffe0 tx=0x7f4bf802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20ea59656fa52c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.615+0000 7f4c00b16640 1 -- 172.21.15.70:0/2230301360 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8035020 con 0x7f4bfc0d0690 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c00b16640 1 -- 172.21.15.70:0/2230301360 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf8031870 con 0x7f4bfc0d0690 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c00b16640 1 -- 172.21.15.70:0/2230301360 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8031c40 con 0x7f4bfc0d0690 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 -- 172.21.15.70:0/2230301360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0d0690 msgr2=0x7f4bfc0d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/2230301360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4bfc06ffe0 tx=0x7f4bf802b910 comp rx=0 tx=0).stop 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 -- 172.21.15.70:0/2230301360 shutdown_connections 2024-04-15T15:15:57.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/2230301360 >> v2:172.21.15.96:3300/0 conn(0x7f4bfc0ae280 0x7f4bfc0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/2230301360 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0d0690 0x7f4bfc0d0a70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 -- 172.21.15.70:0/2230301360 >> 172.21.15.70:0/2230301360 conn(0x7f4bfc06be00 msgr2=0x7f4bfc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 -- 172.21.15.70:0/2230301360 shutdown_connections 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.616+0000 7f4c03d7c640 1 -- 172.21.15.70:0/2230301360 wait complete. 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.617+0000 7f4c03d7c640 1 Processor -- start 2024-04-15T15:15:57.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.617+0000 7f4c03d7c640 1 -- start start 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bfc0b3160 0x7f4bfc15f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bfc0d0690 0x7f4bfc06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc15c870 con 0x7f4bfc0ae280 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4bfc15c590 con 0x7f4bfc0b3160 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c03d7c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4bfc0b4fb0 con 0x7f4bfc0d0690 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43174/0 (socket says 172.21.15.70:43174) 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 -- 172.21.15.70:0/962083041 learned_addr learned my addr 172.21.15.70:0/962083041 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:57.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01317640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bfc0b3160 0x7f4bfc15f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c02319640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bfc0d0690 0x7f4bfc06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 -- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bfc0d0690 msgr2=0x7f4bfc06dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bfc0d0690 0x7f4bfc06dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 -- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bfc0b3160 msgr2=0x7f4bfc15f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bfc0b3160 0x7f4bfc15f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.618+0000 7f4c01b18640 1 -- 172.21.15.70:0/962083041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bfc1f4dc0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4c01b18640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4bf802b950 tx=0x7f4bf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8040070 con 0x7f4bfc0ae280 2024-04-15T15:15:57.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf803a420 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bfc0cacf0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8035040 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.619+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bfc1f8dc0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.620+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bfc0689e0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.620+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4bf804b060 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.621+0000 7f4bf27fc640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 0x7f4be006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.621+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4bf80adfd0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.621+0000 7f4c01317640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 0x7f4be006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.621+0000 7f4c01317640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 0x7f4be006edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.625+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bfc0cacf0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.800+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4bfc15aeb0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.801+0000 7f4bf27fc640 1 -- 172.21.15.70:0/962083041 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f4bfc1f4dc0 con 0x7f4bfc0ae280 2024-04-15T15:15:57.803 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.802+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 msgr2=0x7f4be006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.802+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 0x7f4be006edb0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec008040 comp rx=0 tx=0).stop 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.802+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 msgr2=0x7f4bfc0ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.802+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4bf802b950 tx=0x7f4bf80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 shutdown_connections 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bfc0d0690 0x7f4bfc06dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bfc0b3160 0x7f4bfc15f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4be006c8f0 0x7f4be006edb0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 --2- 172.21.15.70:0/962083041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bfc0ae280 0x7f4bfc0ca7b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 >> 172.21.15.70:0/962083041 conn(0x7f4bfc06be00 msgr2=0x7f4bfc0cc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 shutdown_connections 2024-04-15T15:15:57.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.803+0000 7f4c03d7c640 1 -- 172.21.15.70:0/962083041 wait complete. 2024-04-15T15:15:57.816 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:57.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.919+0000 7fb3b0772640 1 Processor -- start 2024-04-15T15:15:57.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.919+0000 7fb3b0772640 1 -- start start 2024-04-15T15:15:57.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3b0772640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3a80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3b0772640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a80d0fb0 con 0x7fb3a80ae280 2024-04-15T15:15:57.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3b0772640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a815cb50 con 0x7fb3a80b3160 2024-04-15T15:15:57.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3b0772640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a815bda0 con 0x7fb3a80d0690 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3aed0f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb3a80ae280 0x7fb3a80ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57034/0 (socket says 172.21.15.70:57034) 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3aed0f640 1 -- 172.21.15.70:0/938033480 learned_addr learned my addr 172.21.15.70:0/938033480 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3a80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069398630 0 0) 0x7fb3a80d0fb0 con 0x7fb3a80ae280 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.920+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb394005730 con 0x7fb3a80ae280 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3973590820 0 0) 0x7fb394005730 con 0x7fb3a80ae280 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 msgr2=0x7fb3a80d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3a80d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 >> v1:172.21.15.96:6789/0 conn(0x7fb3a80b3160 legacy=0x7fb3a815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a815bfd0 con 0x7fb3a80ae280 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fb3a00019f0 con 0x7fb3a80ae280 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 >> v1:172.21.15.70:6789/0 conn(0x7fb3a80ae280 legacy=0x7fb3a80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb394005bf0 0x7fb394005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.926 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3940065a0 0x7fb394006a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3940072d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb39400dbd0 con 0x7fb394005bf0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb39400dd50 con 0x7fb3940065a0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb39400df50 con 0x7fb3a80d0690 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb394005bf0 0x7fb394005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3add0d640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3940065a0 0x7fb394006a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3aed0f640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3940072d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 -- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 msgr2=0x7fb3940072d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3940072d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 -- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3940065a0 msgr2=0x7fb394006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3940065a0 0x7fb394006a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3ae50e640 1 -- 172.21.15.70:0/938033480 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb394005730 con 0x7fb394005bf0 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.921+0000 7fb3aed0f640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb3a80d0690 0x7fb3940072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:57.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb394005bf0 0x7fb394005fd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009aa0 tx=0x7fb394007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe6dafde2cb788ef server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4039070 con 0x7fb394005bf0 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a402c6b0 con 0x7fb394005bf0 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3ad50c640 1 -- 172.21.15.70:0/938033480 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4009670 con 0x7fb394005bf0 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3b0772640 1 -- 172.21.15.70:0/938033480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb394005bf0 msgr2=0x7fb394005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3b0772640 1 --2- 172.21.15.70:0/938033480 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb394005bf0 0x7fb394005fd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009aa0 tx=0x7fb394007a40 comp rx=0 tx=0).stop 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.922+0000 7fb3aed0f640 1 -- 172.21.15.70:0/938033480 reap_dead start 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.923+0000 7fb3b0772640 1 -- 172.21.15.70:0/938033480 shutdown_connections 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.923+0000 7fb3b0772640 1 -- 172.21.15.70:0/938033480 >> 172.21.15.70:0/938033480 conn(0x7fb3a806be00 msgr2=0x7fb3a806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.923+0000 7fb3b0772640 1 -- 172.21.15.70:0/938033480 shutdown_connections 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.923+0000 7fb3b0772640 1 -- 172.21.15.70:0/938033480 wait complete. 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 Processor -- start 2024-04-15T15:15:57.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 -- start start 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb394005bf0 0x7fb3a8071f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3a815fa90 0x7fb3a8155b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.924+0000 7fb3b0772640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a815c870 con 0x7fb3a81566f0 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3b0772640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a815c590 con 0x7fb3a815fa90 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3b0772640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a80b4fb0 con 0x7fb394005bf0 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43190/0 (socket says 172.21.15.70:43190) 2024-04-15T15:15:57.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 -- 172.21.15.70:0/3379000488 learned_addr learned my addr 172.21.15.70:0/3379000488 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3add0d640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3a815fa90 0x7fb3a8155b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb394005bf0 0x7fb3a8071f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 -- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb394005bf0 msgr2=0x7fb3a8071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb394005bf0 0x7fb3a8071f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 -- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3a815fa90 msgr2=0x7fb3a8155b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3a815fa90 0x7fb3a8155b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 -- 172.21.15.70:0/3379000488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a80aeaa0 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3aed0f640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80d11c0 tx=0x7fb3a000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a000eda0 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a0018070 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0013640 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a80aed80 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.925+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a80af340 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.926+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3a80689e0 con 0x7fb3a81566f0 2024-04-15T15:15:57.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.927+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb3a00137e0 con 0x7fb3a81566f0 2024-04-15T15:15:57.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.927+0000 7fb39effd640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 0x7fb38c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:57.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.927+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb3a008c330 con 0x7fb3a81566f0 2024-04-15T15:15:57.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.927+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 0x7fb38c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:57.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.927+0000 7fb3ae50e640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 0x7fb38c06ede0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009aa0 tx=0x7fb3a4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:57.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:57.931+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb3a005a2d0 con 0x7fb3a81566f0 2024-04-15T15:15:58.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.108+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb3a80739e0 con 0x7fb3a81566f0 2024-04-15T15:15:58.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.108+0000 7fb39effd640 1 -- 172.21.15.70:0/3379000488 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb3a0059bd0 con 0x7fb3a81566f0 2024-04-15T15:15:58.110 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:15:58.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 msgr2=0x7fb38c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 0x7fb38c06ede0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4009aa0 tx=0x7fb3a4030000 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 msgr2=0x7fb3a81560a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fb3a80d11c0 tx=0x7fb3a000e530 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 shutdown_connections 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb38c06c920 0x7fb38c06ede0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3a81566f0 0x7fb3a81560a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3a815fa90 0x7fb3a8155b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 --2- 172.21.15.70:0/3379000488 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb394005bf0 0x7fb3a8071f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 >> 172.21.15.70:0/3379000488 conn(0x7fb3a806be00 msgr2=0x7fb3a81593e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.110+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 shutdown_connections 2024-04-15T15:15:58.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.111+0000 7fb3b0772640 1 -- 172.21.15.70:0/3379000488 wait complete. 2024-04-15T15:15:58.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:15:58.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 Processor -- start 2024-04-15T15:15:58.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 -- start start 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 0x7ff0cc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0cc0b47a0 con 0x7ff0cc15e700 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff0cc0b53b0 con 0x7ff0cc0d2360 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.229+0000 7ff0d2ff4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc0b4a80 con 0x7ff0cc15eae0 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0d0d90640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff0cc15e700 0x7ff0cc0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57046/0 (socket says 172.21.15.70:57046) 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43206/0 (socket says 172.21.15.70:43206) 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0d0d90640 1 -- 172.21.15.70:0/1093130178 learned_addr learned my addr 172.21.15.70:0/1093130178 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0d1591640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 0x7ff0cc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.232 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1093130178 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 msgr2=0x7ff0cc0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 0x7ff0cc0d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1093130178 >> v1:172.21.15.70:6789/0 conn(0x7ff0cc15e700 legacy=0x7ff0cc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1093130178 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc0b4d60 con 0x7ff0cc0d2360 2024-04-15T15:15:58.233 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.230+0000 7ff0d1591640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 0x7ff0cc0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:58.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0cbfff640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80097a0 tx=0x7ff0b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e57aaa181b1c49 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0cb7fe640 1 -- 172.21.15.70:0/1093130178 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8035020 con 0x7ff0cc0d2360 2024-04-15T15:15:58.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0cb7fe640 1 -- 172.21.15.70:0/1093130178 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b8031830 con 0x7ff0cc0d2360 2024-04-15T15:15:58.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0cb7fe640 1 -- 172.21.15.70:0/1093130178 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8031c00 con 0x7ff0cc0d2360 2024-04-15T15:15:58.234 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1093130178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc0d2360 msgr2=0x7ff0cc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80097a0 tx=0x7ff0b802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1093130178 shutdown_connections 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc15eae0 0x7ff0cc0d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1093130178 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc0d2360 0x7ff0cc0d2b60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1093130178 >> 172.21.15.70:0/1093130178 conn(0x7ff0cc08bf60 msgr2=0x7ff0cc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1093130178 shutdown_connections 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.231+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1093130178 wait complete. 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.232+0000 7ff0d2ff4640 1 Processor -- start 2024-04-15T15:15:58.235 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.232+0000 7ff0d2ff4640 1 -- start start 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc0d2360 0x7ff0cc1605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 0x7ff0cc1f3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc0d1d40 con 0x7ff0cc15e700 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0cc0b4450 con 0x7ff0cc15eae0 2024-04-15T15:15:58.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0d2ff4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff0cc0d1ec0 con 0x7ff0cc0d2360 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0cbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0cbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43208/0 (socket says 172.21.15.70:43208) 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.233+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1702419385 learned_addr learned my addr 172.21.15.70:0/1702419385 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0d0d90640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc0d2360 0x7ff0cc1605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0d1591640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 0x7ff0cc1f3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc0d2360 msgr2=0x7ff0cc1605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc0d2360 0x7ff0cc1605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 msgr2=0x7ff0cc1f3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 0x7ff0cc1f3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 -- 172.21.15.70:0/1702419385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0cc1f8b40 con 0x7ff0cc15e700 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0d1591640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 0x7ff0cc1f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0cbfff640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0b802bee0 tx=0x7ff0b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8041070 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b803a950 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0cc1f4480 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8035040 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.234+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0cc15f810 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.235+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0cc0689e0 con 0x7ff0cc15e700 2024-04-15T15:15:58.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.236+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0b8038020 con 0x7ff0cc15e700 2024-04-15T15:15:58.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.236+0000 7ff0c97fa640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.236+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff0b80ad980 con 0x7ff0cc15e700 2024-04-15T15:15:58.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.236+0000 7ff0d0d90640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.237+0000 7ff0d0d90640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 0x7ff0b006edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0001000 tx=0x7ff0c0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.240+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0cc1f8b40 con 0x7ff0cc15e700 2024-04-15T15:15:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.419+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff0cc0ba500 con 0x7ff0cc15e700 2024-04-15T15:15:58.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.419+0000 7ff0c97fa640 1 -- 172.21.15.70:0/1702419385 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7ff0b807b360 con 0x7ff0cc15e700 2024-04-15T15:15:58.421 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 msgr2=0x7ff0b006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 0x7ff0b006edb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff0c0001000 tx=0x7ff0c0009040 comp rx=0 tx=0).stop 2024-04-15T15:15:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 msgr2=0x7ff0cc1f8600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff0b802bee0 tx=0x7ff0b80046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 shutdown_connections 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0cc15eae0 0x7ff0cc1f3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0b006c8f0 0x7ff0b006edb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0cc15e700 0x7ff0cc1f8600 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 --2- 172.21.15.70:0/1702419385 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0cc0d2360 0x7ff0cc1605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 >> 172.21.15.70:0/1702419385 conn(0x7ff0cc08bf60 msgr2=0x7ff0cc0d05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 shutdown_connections 2024-04-15T15:15:58.424 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.421+0000 7ff0d2ff4640 1 -- 172.21.15.70:0/1702419385 wait complete. 2024-04-15T15:15:58.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:15:58.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.537+0000 7f38e3488640 1 Processor -- start 2024-04-15T15:15:58.539 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.537+0000 7f38e3488640 1 -- start start 2024-04-15T15:15:58.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e3488640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f38dc0b3160 0x7f38dc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.540 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e3488640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f38dc0ae280 0x7f38dc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e3488640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38dc15cb50 con 0x7f38dc0d0690 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e3488640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f38dc15bfd0 con 0x7f38dc0b3160 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e3488640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f38dc15c2b0 con 0x7f38dc0ae280 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e0a23640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f38dc0b3160 0x7f38dc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e0a23640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f38dc0b3160 0x7f38dc15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43224/0 (socket says 172.21.15.70:43224) 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e0a23640 1 -- 172.21.15.70:0/2249071 learned_addr learned my addr 172.21.15.70:0/2249071 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.538+0000 7f38e1a25640 1 --2- 172.21.15.70:0/2249071 >> v2:172.21.15.96:3300/0 conn(0x7f38dc0ae280 0x7f38dc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38e0a23640 1 -- 172.21.15.70:0/2249071 >> v2:172.21.15.96:3300/0 conn(0x7f38dc0ae280 msgr2=0x7f38dc0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38e0a23640 1 --2- 172.21.15.70:0/2249071 >> v2:172.21.15.96:3300/0 conn(0x7f38dc0ae280 0x7f38dc0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38e0a23640 1 -- 172.21.15.70:0/2249071 >> v1:172.21.15.96:6789/0 conn(0x7f38dc0d0690 legacy=0x7f38dc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.541 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38e0a23640 1 -- 172.21.15.70:0/2249071 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38dc15c590 con 0x7f38dc0b3160 2024-04-15T15:15:58.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38e0a23640 1 --2- 172.21.15.70:0/2249071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc15b690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f38dc06ffe0 tx=0x7f38cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2171e036f0a758db server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38d3fff640 1 -- 172.21.15.70:0/2249071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc034070 con 0x7f38dc0b3160 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38d3fff640 1 -- 172.21.15.70:0/2249071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38cc030860 con 0x7f38dc0b3160 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.539+0000 7f38d3fff640 1 -- 172.21.15.70:0/2249071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc030c30 con 0x7f38dc0b3160 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 -- 172.21.15.70:0/2249071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 msgr2=0x7f38dc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 --2- 172.21.15.70:0/2249071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc15b690 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f38dc06ffe0 tx=0x7f38cc02b9a0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 -- 172.21.15.70:0/2249071 shutdown_connections 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 --2- 172.21.15.70:0/2249071 >> v2:172.21.15.96:3300/0 conn(0x7f38dc0ae280 0x7f38dc0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 --2- 172.21.15.70:0/2249071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc15b690 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 -- 172.21.15.70:0/2249071 >> 172.21.15.70:0/2249071 conn(0x7f38dc06be00 msgr2=0x7f38dc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 -- 172.21.15.70:0/2249071 shutdown_connections 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.540+0000 7f38e3488640 1 -- 172.21.15.70:0/2249071 wait complete. 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.541+0000 7f38e3488640 1 Processor -- start 2024-04-15T15:15:58.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.541+0000 7f38e3488640 1 -- start start 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38dc0ae280 0x7f38dc0cbb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f38dc0d0690 0x7f38dc1f4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc15bbd0 con 0x7f38dc0b3160 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f38dc0b4fb0 con 0x7f38dc0ae280 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e3488640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f38dc15bd50 con 0x7f38dc0d0690 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43232/0 (socket says 172.21.15.70:43232) 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 -- 172.21.15.70:0/2507133364 learned_addr learned my addr 172.21.15.70:0/2507133364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e1224640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38dc0ae280 0x7f38dc0cbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e1a25640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f38dc0d0690 0x7f38dc1f4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 -- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f38dc0d0690 msgr2=0x7f38dc1f4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f38dc0d0690 0x7f38dc1f4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 -- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38dc0ae280 msgr2=0x7f38dc0cbb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38dc0ae280 0x7f38dc0cbb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.542+0000 7f38e0a23640 1 -- 172.21.15.70:0/2507133364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38dc1f8550 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38e0a23640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f38cc02b9e0 tx=0x7f38cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc034040 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38cc039d90 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38cc043610 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38dc1f5a90 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.543+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38dc1f8810 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.544+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38dc0689e0 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.544+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f38cc04a060 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.545+0000 7f38d1ffb640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 0x7f38c006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.545+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f38cc0adfc0 con 0x7f38dc0b3160 2024-04-15T15:15:58.550 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.545+0000 7f38e1224640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 0x7f38c006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:58.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.545+0000 7f38e1224640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 0x7f38c006edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38d8007fd0 tx=0x7f38d8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:58.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.549+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f38cc03e070 con 0x7f38dc0b3160 2024-04-15T15:15:58.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.736+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f38dc15ae40 con 0x7f38dc0b3160 2024-04-15T15:15:58.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.736+0000 7f38d1ffb640 1 -- 172.21.15.70:0/2507133364 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f38cc037360 con 0x7f38dc0b3160 2024-04-15T15:15:58.738 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 msgr2=0x7f38c006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 0x7f38c006edb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f38d8007fd0 tx=0x7f38d8009040 comp rx=0 tx=0).stop 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 msgr2=0x7f38dc1f9010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f38cc02b9e0 tx=0x7f38cc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 shutdown_connections 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f38dc0d0690 0x7f38dc1f4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f38c006c8f0 0x7f38c006edb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f38dc0b3160 0x7f38dc1f9010 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 --2- 172.21.15.70:0/2507133364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f38dc0ae280 0x7f38dc0cbb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:58.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 >> 172.21.15.70:0/2507133364 conn(0x7f38dc06be00 msgr2=0x7f38dc156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:58.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 shutdown_connections 2024-04-15T15:15:58.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:58.738+0000 7f38e3488640 1 -- 172.21.15.70:0/2507133364 wait complete. 2024-04-15T15:15:58.751 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:15:58.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:58.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:15:58.780 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:15:58.784 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:15:58.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:15:58.784 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:15:58.836 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:15:58.836 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:58.855 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:15:58.856 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:15:58.915 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:15:59.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.016+0000 7f1bf96df640 1 Processor -- start 2024-04-15T15:15:59.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.016+0000 7f1bf96df640 1 -- start start 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf96df640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1bf415e9a0 0x7f1bf40d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf96df640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1bf40d2ba0 0x7f1bf40d2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf96df640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40d3510 con 0x7f1bf40d1400 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf96df640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf40d1e60 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf96df640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1bf40d18a0 con 0x7f1bf415e9a0 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1bf40d2ba0 0x7f1bf40d2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf27fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1bf40d1400 0x7f1bf40cf210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57054/0 (socket says 172.21.15.70:57054) 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.017+0000 7f1bf27fc640 1 -- 172.21.15.70:0/573834971 learned_addr learned my addr 172.21.15.70:0/573834971 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/573834971 >> v2:172.21.15.96:3300/0 conn(0x7f1bf415e9a0 0x7f1bf40d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf37fe640 1 -- 172.21.15.70:0/573834971 >> v2:172.21.15.96:3300/0 conn(0x7f1bf415e9a0 msgr2=0x7f1bf40d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/573834971 >> v2:172.21.15.96:3300/0 conn(0x7f1bf415e9a0 0x7f1bf40d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf37fe640 1 -- 172.21.15.70:0/573834971 >> v1:172.21.15.70:6789/0 conn(0x7f1bf40d1400 legacy=0x7f1bf40cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf37fe640 1 -- 172.21.15.70:0/573834971 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40d1b80 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf1ffb640 1 -- 172.21.15.70:0/573834971 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2716163815 0 0) 0x7f1be8001110 con 0x7f1bf40d1400 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.018+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/573834971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf40d2fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bf408dfc0 tx=0x7f1bec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3725636c92affeba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf1ffb640 1 -- 172.21.15.70:0/573834971 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec035020 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf1ffb640 1 -- 172.21.15.70:0/573834971 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bec031830 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf1ffb640 1 -- 172.21.15.70:0/573834971 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec031c00 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 -- 172.21.15.70:0/573834971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 msgr2=0x7f1bf40d2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 --2- 172.21.15.70:0/573834971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf40d2fd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bf408dfc0 tx=0x7f1bec02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 -- 172.21.15.70:0/573834971 shutdown_connections 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 --2- 172.21.15.70:0/573834971 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf40d2fd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 --2- 172.21.15.70:0/573834971 >> v2:172.21.15.96:3300/0 conn(0x7f1bf415e9a0 0x7f1bf40d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 -- 172.21.15.70:0/573834971 >> 172.21.15.70:0/573834971 conn(0x7f1bf409d950 msgr2=0x7f1bf408bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 -- 172.21.15.70:0/573834971 shutdown_connections 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.019+0000 7f1bf96df640 1 -- 172.21.15.70:0/573834971 wait complete. 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.020+0000 7f1bf96df640 1 Processor -- start 2024-04-15T15:15:59.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.020+0000 7f1bf96df640 1 -- start start 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1bf40d1400 0x7f1bf40b7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 0x7f1bf41611e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf40cf920 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1bf406d2d0 con 0x7f1bf415e9a0 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf96df640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1bf40d2360 con 0x7f1bf40d1400 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf27fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43250/0 (socket says 172.21.15.70:43250) 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.021+0000 7f1bf27fc640 1 -- 172.21.15.70:0/3421326554 learned_addr learned my addr 172.21.15.70:0/3421326554 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 0x7f1bf41611e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1bf40d1400 0x7f1bf40b7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 -- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1bf40d1400 msgr2=0x7f1bf40b7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1bf40d1400 0x7f1bf40b7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 -- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 msgr2=0x7f1bf41611e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 0x7f1bf41611e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 -- 172.21.15.70:0/3421326554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf40cc5e0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf37fe640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 0x7f1bf41611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf27fc640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1be800c780 tx=0x7f1be800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be8018070 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1be80072d0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf415efd0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1be80047e0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.022+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bf415f590 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.023+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bf40690f0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.023+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1be801b070 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.024+0000 7f1bdb7fe640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 0x7f1be006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.024+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1be808b200 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.024+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 0x7f1be006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.024+0000 7f1bf2ffd640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 0x7f1be006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1be40099c0 tx=0x7f1be4003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.028+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1bf40d3510 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.203+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1bf406cd30 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.205+0000 7f1bdb7fe640 1 -- 172.21.15.70:0/3421326554 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7f1be80592c0 con 0x7f1bf40d2ba0 2024-04-15T15:15:59.207 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:15:59.207 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 msgr2=0x7f1be006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 0x7f1be006edb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1be40099c0 tx=0x7f1be4003990 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 msgr2=0x7f1bf415c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1be800c780 tx=0x7f1be800cc40 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 shutdown_connections 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1bf415e9a0 0x7f1bf41611e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1be006c8f0 0x7f1be006edb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1bf40d2ba0 0x7f1bf415c860 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 --2- 172.21.15.70:0/3421326554 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1bf40d1400 0x7f1bf40b7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 >> 172.21.15.70:0/3421326554 conn(0x7f1bf409d950 msgr2=0x7f1bf40cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 shutdown_connections 2024-04-15T15:15:59.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.207+0000 7f1bf96df640 1 -- 172.21.15.70:0/3421326554 wait complete. 2024-04-15T15:15:59.220 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:15:59.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:15:59.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.324+0000 7f70a8681640 1 Processor -- start 2024-04-15T15:15:59.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.324+0000 7f70a8681640 1 -- start start 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.325+0000 7f70a8681640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70a015ebb0 0x7f70a006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.325+0000 7f70a8681640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a00ced60 con 0x7f70a009b780 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.325+0000 7f70a8681640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70a00cccb0 con 0x7f70a009a9d0 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.325+0000 7f70a8681640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f70a00cd8c0 con 0x7f70a015ebb0 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a5c1c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f70a015ebb0 0x7f70a006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a6c1e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f70a009b780 0x7f70a00b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57062/0 (socket says 172.21.15.70:57062) 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a6c1e640 1 -- 172.21.15.70:0/246524508 learned_addr learned my addr 172.21.15.70:0/246524508 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a5c1c640 1 -- 172.21.15.70:0/246524508 >> v1:172.21.15.96:6789/0 conn(0x7f70a009a9d0 legacy=0x7f70a009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a5c1c640 1 -- 172.21.15.70:0/246524508 >> v1:172.21.15.70:6789/0 conn(0x7f70a009b780 legacy=0x7f70a00b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a541b640 1 -- 172.21.15.70:0/246524508 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1283550610 0 0) 0x7f7098001490 con 0x7f70a009b780 2024-04-15T15:15:59.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.326+0000 7f70a5c1c640 1 -- 172.21.15.70:0/246524508 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00ccf90 con 0x7f70a015ebb0 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a5c1c640 1 --2- 172.21.15.70:0/246524508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a015ebb0 0x7f70a006cd00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f70900097a0 tx=0x7f709002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dff59507684b540 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a541b640 1 -- 172.21.15.70:0/246524508 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7090034070 con 0x7f70a015ebb0 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a541b640 1 -- 172.21.15.70:0/246524508 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7090030610 con 0x7f70a015ebb0 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a541b640 1 -- 172.21.15.70:0/246524508 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70900309e0 con 0x7f70a015ebb0 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 -- 172.21.15.70:0/246524508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a015ebb0 msgr2=0x7f70a006cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 --2- 172.21.15.70:0/246524508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a015ebb0 0x7f70a006cd00 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f70900097a0 tx=0x7f709002b960 comp rx=0 tx=0).stop 2024-04-15T15:15:59.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 -- 172.21.15.70:0/246524508 shutdown_connections 2024-04-15T15:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 --2- 172.21.15.70:0/246524508 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a015ebb0 0x7f70a006cd00 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 -- 172.21.15.70:0/246524508 >> 172.21.15.70:0/246524508 conn(0x7f70a00b3fb0 msgr2=0x7f70a00b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 -- 172.21.15.70:0/246524508 shutdown_connections 2024-04-15T15:15:59.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.327+0000 7f70a8681640 1 -- 172.21.15.70:0/246524508 wait complete. 2024-04-15T15:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 Processor -- start 2024-04-15T15:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 -- start start 2024-04-15T15:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f70a009b780 0x7f70a00c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 0x7f70a01f7090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f70a006d240 con 0x7f70a009a9d0 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f70a00cd550 con 0x7f70a00c9ad0 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a8681640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f70a00b5310 con 0x7f70a009b780 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a641d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a641d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43258/0 (socket says 172.21.15.70:43258) 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.329+0000 7f70a641d640 1 -- 172.21.15.70:0/3260493158 learned_addr learned my addr 172.21.15.70:0/3260493158 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a5c1c640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f70a009b780 0x7f70a00c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 -- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f70a009b780 msgr2=0x7f70a00c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a6c1e640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 0x7f70a01f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f70a009b780 0x7f70a00c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 -- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 msgr2=0x7f70a01f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 0x7f70a01f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 -- 172.21.15.70:0/3260493158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a00ca0e0 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a6c1e640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 0x7f70a01f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a641d640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c450 tx=0x7f709c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c0052d0 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f709c005470 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a00ca3c0 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f709c012870 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.330+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a00ca980 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.331+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a00690f0 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.332+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f709c012a10 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.332+0000 7f7096ffd640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 0x7f708406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.332+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f709c08bb50 con 0x7f70a009a9d0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.332+0000 7f70a5c1c640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 0x7f708406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.332+0000 7f70a5c1c640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 0x7f708406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7090036000 tx=0x7f7090008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.336+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f709c059b80 con 0x7f70a009a9d0 2024-04-15T15:15:59.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.513+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f70a00cf070 con 0x7f70a009a9d0 2024-04-15T15:15:59.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.514+0000 7f7096ffd640 1 -- 172.21.15.70:0/3260493158 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f709c059480 con 0x7f70a009a9d0 2024-04-15T15:15:59.516 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 msgr2=0x7f708406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 0x7f708406edb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7090036000 tx=0x7f7090008030 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 msgr2=0x7f70a00c7fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f709c00c450 tx=0x7f709c00c910 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 shutdown_connections 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f70a00c9ad0 0x7f70a01f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f70a009b780 0x7f70a00c9590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f708406c8f0 0x7f708406edb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 --2- 172.21.15.70:0/3260493158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70a009a9d0 0x7f70a00c7fd0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 >> 172.21.15.70:0/3260493158 conn(0x7f70a00b3fb0 msgr2=0x7f70a00700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 shutdown_connections 2024-04-15T15:15:59.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.516+0000 7f70a8681640 1 -- 172.21.15.70:0/3260493158 wait complete. 2024-04-15T15:15:59.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:15:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.633+0000 7f0b8daa5640 1 Processor -- start 2024-04-15T15:15:59.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.633+0000 7f0b8daa5640 1 -- start start 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b8daa5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b8815ebb0 0x7f0b8806cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b8daa5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b880ced60 con 0x7f0b8809a9d0 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b8daa5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0b880cccb0 con 0x7f0b8809b780 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b8daa5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0b880cd8c0 con 0x7f0b8815ebb0 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b877fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0b8809a9d0 0x7f0b8809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57074/0 (socket says 172.21.15.70:57074) 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b877fe640 1 -- 172.21.15.70:0/2067618461 learned_addr learned my addr 172.21.15.70:0/2067618461 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b7ffff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b8815ebb0 0x7f0b8806cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 464643806 0 0) 0x7f0b78001110 con 0x7f0b8809a9d0 2024-04-15T15:15:59.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0b74003520 con 0x7f0b8809a9d0 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2569835726 0 0) 0x7f0b74003520 con 0x7f0b8809a9d0 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b8815ebb0 msgr2=0x7f0b8806cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b8815ebb0 0x7f0b8806cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 >> v1:172.21.15.96:6789/0 conn(0x7f0b8809b780 legacy=0x7f0b880b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b880ccf90 con 0x7f0b8809a9d0 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f0b78001840 con 0x7f0b8809a9d0 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 >> v1:172.21.15.70:6789/0 conn(0x7f0b8809a9d0 legacy=0x7f0b8809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b740039e0 0x7f0b74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b74004300 0x7f0b74004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b74004f90 0x7f0b7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880cd8c0 con 0x7f0b74004f90 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0b7400bf00 con 0x7f0b74004300 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0b7400c080 con 0x7f0b740039e0 2024-04-15T15:15:59.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b87fff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b74004f90 0x7f0b7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.634+0000 7f0b877fe640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b740039e0 0x7f0b74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b7ffff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b74004300 0x7f0b74004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b87fff640 1 -- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b740039e0 msgr2=0x7f0b74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b87fff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b740039e0 0x7f0b74003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b87fff640 1 -- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b74004300 msgr2=0x7f0b74004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b87fff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b74004300 0x7f0b74004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.635+0000 7f0b87fff640 1 -- 172.21.15.70:0/2067618461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b74003520 con 0x7f0b74004f90 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b87fff640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b74004f90 0x7f0b7400b9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0b80001f30 tx=0x7f0b74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126ff69c9f913d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80001b30 con 0x7f0b74004f90 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8002d6e0 con 0x7f0b74004f90 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b86ffd640 1 -- 172.21.15.70:0/2067618461 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8002dad0 con 0x7f0b74004f90 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 -- 172.21.15.70:0/2067618461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b74004f90 msgr2=0x7f0b7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/2067618461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b74004f90 0x7f0b7400b9c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0b80001f30 tx=0x7f0b74005830 comp rx=0 tx=0).stop 2024-04-15T15:15:59.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b87fff640 1 -- 172.21.15.70:0/2067618461 reap_dead start 2024-04-15T15:15:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 -- 172.21.15.70:0/2067618461 shutdown_connections 2024-04-15T15:15:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 -- 172.21.15.70:0/2067618461 >> 172.21.15.70:0/2067618461 conn(0x7f0b880b3fb0 msgr2=0x7f0b880b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 -- 172.21.15.70:0/2067618461 shutdown_connections 2024-04-15T15:15:59.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.636+0000 7f0b8daa5640 1 -- 172.21.15.70:0/2067618461 wait complete. 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.637+0000 7f0b8daa5640 1 Processor -- start 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 -- start start 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b88154f60 0x7f0b8815fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 0x7f0b880c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880b5310 con 0x7f0b880ca1d0 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0b880cd550 con 0x7f0b881603a0 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b8daa5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0b880b5490 con 0x7f0b88154f60 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b87fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43290/0 (socket says 172.21.15.70:43290) 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.638+0000 7f0b87fff640 1 -- 172.21.15.70:0/3960497839 learned_addr learned my addr 172.21.15.70:0/3960497839 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b7ffff640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 0x7f0b880c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b877fe640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b88154f60 0x7f0b8815fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 -- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b88154f60 msgr2=0x7f0b8815fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b88154f60 0x7f0b8815fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 -- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 msgr2=0x7f0b880c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 0x7f0b880c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 -- 172.21.15.70:0/3960497839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b8815edc0 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b7ffff640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 0x7f0b880c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b87fff640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0b8002c120 tx=0x7f0b800015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80001b30 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b80045070 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b80003b90 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b8815f0a0 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.639+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b8815f660 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.640+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b88160780 con 0x7f0b880ca1d0 2024-04-15T15:15:59.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.641+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0b80005070 con 0x7f0b880ca1d0 2024-04-15T15:15:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.641+0000 7f0b84ff9640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 0x7f0b6806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.641+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0b800addf0 con 0x7f0b880ca1d0 2024-04-15T15:15:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.641+0000 7f0b877fe640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 0x7f0b6806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.641+0000 7f0b877fe640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 0x7f0b6806ede0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b780012b0 tx=0x7f0b78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.645+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0b88160780 con 0x7f0b880ca1d0 2024-04-15T15:15:59.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.819+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0b880690f0 con 0x7f0b880ca1d0 2024-04-15T15:15:59.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.819+0000 7f0b84ff9640 1 -- 172.21.15.70:0/3960497839 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f0b8007bd90 con 0x7f0b880ca1d0 2024-04-15T15:15:59.821 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:15:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 msgr2=0x7f0b6806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 0x7f0b6806ede0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b780012b0 tx=0x7f0b78009040 comp rx=0 tx=0).stop 2024-04-15T15:15:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 msgr2=0x7f0b880c7870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0b8002c120 tx=0x7f0b800015b0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 shutdown_connections 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0b6806c920 0x7f0b6806ede0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0b880ca1d0 0x7f0b880c7870 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0b881603a0 0x7f0b880c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 --2- 172.21.15.70:0/3960497839 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0b88154f60 0x7f0b8815fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 >> 172.21.15.70:0/3960497839 conn(0x7f0b880b3fb0 msgr2=0x7f0b8806d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 shutdown_connections 2024-04-15T15:15:59.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.821+0000 7f0b8daa5640 1 -- 172.21.15.70:0/3960497839 wait complete. 2024-04-15T15:15:59.834 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:15:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.938+0000 7f0348de6640 1 Processor -- start 2024-04-15T15:15:59.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.938+0000 7f0348de6640 1 -- start start 2024-04-15T15:15:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.939+0000 7f0348de6640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.939+0000 7f0348de6640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f034409b0e0 0x7f03440b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.939+0000 7f0348de6640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03440d0350 con 0x7f034409b650 2024-04-15T15:15:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.939+0000 7f0348de6640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f03440b5b00 con 0x7f034409b0e0 2024-04-15T15:15:59.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.939+0000 7f0348de6640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f03440ce100 con 0x7f034415c8d0 2024-04-15T15:15:59.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f034259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f0342d9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f034409b0e0 0x7f03440b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f034259c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:53072/0 (socket says 172.21.15.70:53072) 2024-04-15T15:15:59.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f034259c640 1 -- 172.21.15.70:0/253866609 learned_addr learned my addr 172.21.15.70:0/253866609 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:59.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f0342d9d640 1 -- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 msgr2=0x7f03440cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:15:59.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f0342d9d640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f0342d9d640 1 -- 172.21.15.70:0/253866609 >> v1:172.21.15.96:6789/0 conn(0x7f034409b650 legacy=0x7f034409a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f0342d9d640 1 -- 172.21.15.70:0/253866609 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03440b54b0 con 0x7f034409b0e0 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.940+0000 7f034259c640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.941+0000 7f0342d9d640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034409b0e0 0x7f03440b5f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f033c009d90 tx=0x7f033c02be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa3ae9ad7a9f2776 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.941+0000 7f034159a640 1 -- 172.21.15.70:0/253866609 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c039070 con 0x7f034409b0e0 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.941+0000 7f034159a640 1 -- 172.21.15.70:0/253866609 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f033c02cbc0 con 0x7f034409b0e0 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.941+0000 7f034159a640 1 -- 172.21.15.70:0/253866609 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033c034440 con 0x7f034409b0e0 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 -- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034409b0e0 msgr2=0x7f03440b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034409b0e0 0x7f03440b5f90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f033c009d90 tx=0x7f033c02be10 comp rx=0 tx=0).stop 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 -- 172.21.15.70:0/253866609 shutdown_connections 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f034409b0e0 0x7f03440b5f90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 --2- 172.21.15.70:0/253866609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034415c8d0 0x7f03440cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 -- 172.21.15.70:0/253866609 >> 172.21.15.70:0/253866609 conn(0x7f03440b3fb0 msgr2=0x7f03440b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 -- 172.21.15.70:0/253866609 shutdown_connections 2024-04-15T15:15:59.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.942+0000 7f0348de6640 1 -- 172.21.15.70:0/253866609 wait complete. 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.943+0000 7f0348de6640 1 Processor -- start 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.943+0000 7f0348de6640 1 -- start start 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 0x7f03440cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 0x7f03441f7f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f03440b6650 con 0x7f034409b0e0 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f03440b64d0 con 0x7f03440cb160 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f03440b67d0 con 0x7f034409b650 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43302/0 (socket says 172.21.15.70:43302) 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 -- 172.21.15.70:0/2335512219 learned_addr learned my addr 172.21.15.70:0/2335512219 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0342d9d640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 0x7f03441f7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0341d9b640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 0x7f03440cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 -- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 msgr2=0x7f03440cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 0x7f03440cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 -- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 msgr2=0x7f03441f7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 0x7f03441f7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 -- 172.21.15.70:0/2335512219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f034415b2b0 con 0x7f034409b0e0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0342d9d640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 0x7f03441f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0341d9b640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 0x7f03440cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f034259c640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f033400e9b0 tx=0x7f033400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0334017070 con 0x7f034409b0e0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f033400c9a0 con 0x7f034409b0e0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f034406cc00 con 0x7f034409b0e0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.944+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f033400cd50 con 0x7f034409b0e0 2024-04-15T15:15:59.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.945+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f034406d1c0 con 0x7f034409b0e0 2024-04-15T15:15:59.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.945+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03440690f0 con 0x7f034409b0e0 2024-04-15T15:15:59.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.946+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f033401a070 con 0x7f034409b0e0 2024-04-15T15:15:59.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.946+0000 7f0332ffd640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 0x7f032406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:15:59.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.947+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f033408bd50 con 0x7f034409b0e0 2024-04-15T15:15:59.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.947+0000 7f0341d9b640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 0x7f032406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:15:59.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.947+0000 7f0341d9b640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 0x7f032406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0338007fd0 tx=0x7f0338009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:15:59.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:15:59.951+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f033408f030 con 0x7f034409b0e0 2024-04-15T15:16:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.126+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f034415c0d0 con 0x7f034409b0e0 2024-04-15T15:16:00.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.126+0000 7f0332ffd640 1 -- 172.21.15.70:0/2335512219 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f0334059e10 con 0x7f034409b0e0 2024-04-15T15:16:00.128 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 msgr2=0x7f032406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 0x7f032406edb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0338007fd0 tx=0x7f0338009040 comp rx=0 tx=0).stop 2024-04-15T15:16:00.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 msgr2=0x7f034415ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f033400e9b0 tx=0x7f033400ee70 comp rx=0 tx=0).stop 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 shutdown_connections 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f03440cb160 0x7f03441f7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f034409b650 0x7f03440cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f032406c8f0 0x7f032406edb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 --2- 172.21.15.70:0/2335512219 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f034409b0e0 0x7f034415ad70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 >> 172.21.15.70:0/2335512219 conn(0x7f03440b3fb0 msgr2=0x7f0344071240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 shutdown_connections 2024-04-15T15:16:00.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.128+0000 7f0348de6640 1 -- 172.21.15.70:0/2335512219 wait complete. 2024-04-15T15:16:00.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:00.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.246+0000 7f27f55b5640 1 Processor -- start 2024-04-15T15:16:00.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.246+0000 7f27f55b5640 1 -- start start 2024-04-15T15:16:00.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27f55b5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f27f015c8d0 0x7f27f00cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27f55b5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f00d0350 con 0x7f27f009b650 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27f55b5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27f00ce100 con 0x7f27f009b0e0 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27f55b5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f27f00b5b00 con 0x7f27f015c8d0 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27eeffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f27f015c8d0 0x7f27f00cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27ee7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f27f009b650 0x7f27f009a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57086/0 (socket says 172.21.15.70:57086) 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.247+0000 7f27ee7fc640 1 -- 172.21.15.70:0/4094055576 learned_addr learned my addr 172.21.15.70:0/4094055576 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.248+0000 7f27eeffd640 1 -- 172.21.15.70:0/4094055576 >> v1:172.21.15.96:6789/0 conn(0x7f27f009b0e0 legacy=0x7f27f00b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.248+0000 7f27eeffd640 1 -- 172.21.15.70:0/4094055576 >> v1:172.21.15.70:6789/0 conn(0x7f27f009b650 legacy=0x7f27f009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.248+0000 7f27edffb640 1 -- 172.21.15.70:0/4094055576 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2117040904 0 0) 0x7f27f00d0350 con 0x7f27f009b650 2024-04-15T15:16:00.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.248+0000 7f27eeffd640 1 -- 172.21.15.70:0/4094055576 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f00b54b0 con 0x7f27f015c8d0 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.248+0000 7f27eeffd640 1 --2- 172.21.15.70:0/4094055576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f015c8d0 0x7f27f00cda70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c62fe9d3cf552d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27edffb640 1 -- 172.21.15.70:0/4094055576 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e00315a0 con 0x7f27f015c8d0 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27edffb640 1 -- 172.21.15.70:0/4094055576 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e0037070 con 0x7f27f015c8d0 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27edffb640 1 -- 172.21.15.70:0/4094055576 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0031960 con 0x7f27f015c8d0 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 -- 172.21.15.70:0/4094055576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f015c8d0 msgr2=0x7f27f00cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 --2- 172.21.15.70:0/4094055576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f015c8d0 0x7f27f00cda70 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e002b840 comp rx=0 tx=0).stop 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 -- 172.21.15.70:0/4094055576 shutdown_connections 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 --2- 172.21.15.70:0/4094055576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f015c8d0 0x7f27f00cda70 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 -- 172.21.15.70:0/4094055576 >> 172.21.15.70:0/4094055576 conn(0x7f27f00b3fb0 msgr2=0x7f27f00b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 -- 172.21.15.70:0/4094055576 shutdown_connections 2024-04-15T15:16:00.252 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.249+0000 7f27f55b5640 1 -- 172.21.15.70:0/4094055576 wait complete. 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.250+0000 7f27f55b5640 1 Processor -- start 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 -- start start 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f27f009b650 0x7f27f0154da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27f009f1c0 0x7f27f00d09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27f00b6820 con 0x7f27f009b0e0 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f27f00b66a0 con 0x7f27f009f1c0 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27f55b5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f27f00b69a0 con 0x7f27f009b650 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27eeffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43332/0 (socket says 172.21.15.70:43332) 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27eeffd640 1 -- 172.21.15.70:0/641271032 learned_addr learned my addr 172.21.15.70:0/641271032 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27ee7fc640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f27f009b650 0x7f27f0154da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.251+0000 7f27ef7fe640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27f009f1c0 0x7f27f00d09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 -- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f27f009b650 msgr2=0x7f27f0154da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f27f009b650 0x7f27f0154da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 -- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27f009f1c0 msgr2=0x7f27f00d09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27f009f1c0 0x7f27f00d09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 -- 172.21.15.70:0/641271032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27f009d420 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27eeffd640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0004230 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27e00043f0 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27f009d700 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27e0032880 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.252+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27f009dcc0 con 0x7f27f009b0e0 2024-04-15T15:16:00.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.253+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27f00690f0 con 0x7f27f009b0e0 2024-04-15T15:16:00.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.253+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f27e0032a20 con 0x7f27f009b0e0 2024-04-15T15:16:00.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.254+0000 7f27d77fe640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.254+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f27e002f030 con 0x7f27f009b0e0 2024-04-15T15:16:00.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.254+0000 7f27ee7fc640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.254+0000 7f27ee7fc640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 0x7f27dc06edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27f0071710 tx=0x7f27e4007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.258+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f27e0041070 con 0x7f27f009b0e0 2024-04-15T15:16:00.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.432+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f27f00cecb0 con 0x7f27f009b0e0 2024-04-15T15:16:00.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.432+0000 7f27d77fe640 1 -- 172.21.15.70:0/641271032 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f27e0044340 con 0x7f27f009b0e0 2024-04-15T15:16:00.434 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 msgr2=0x7f27dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 0x7f27dc06edb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f27f0071710 tx=0x7f27e4007b50 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 msgr2=0x7f27f009ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f27e00097a0 tx=0x7f27e00046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 shutdown_connections 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f27f009f1c0 0x7f27f00d09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f27f009b650 0x7f27f0154da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f27dc06c8f0 0x7f27dc06edb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 --2- 172.21.15.70:0/641271032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f27f009b0e0 0x7f27f009ccd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 >> 172.21.15.70:0/641271032 conn(0x7f27f00b3fb0 msgr2=0x7f27f00b6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 shutdown_connections 2024-04-15T15:16:00.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.434+0000 7f27f55b5640 1 -- 172.21.15.70:0/641271032 wait complete. 2024-04-15T15:16:00.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:00.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.550+0000 7f37fb1bd640 1 Processor -- start 2024-04-15T15:16:00.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.550+0000 7f37fb1bd640 1 -- start start 2024-04-15T15:16:00.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37fb1bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37f4072e30 0x7f37f4165b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37fb1bd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f40b4190 con 0x7f37f4158770 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37fb1bd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f40b4470 con 0x7f37f40b5230 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37fb1bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37f40b4da0 con 0x7f37f4072e30 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37f8f59640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f37f4158770 0x7f37f415bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57094/0 (socket says 172.21.15.70:57094) 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37f8f59640 1 -- 172.21.15.70:0/955063226 learned_addr learned my addr 172.21.15.70:0/955063226 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37f4072e30 0x7f37f4165b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1318752568 0 0) 0x7f37f40b4190 con 0x7f37f4158770 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37dc004630 con 0x7f37f4158770 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1447938520 0 0) 0x7f37f40b4470 con 0x7f37f40b5230 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37f40b4190 con 0x7f37f40b5230 2024-04-15T15:16:00.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 976177873 0 0) 0x7f37dc004630 con 0x7f37f4158770 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37f4072e30 msgr2=0x7f37f4165b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37f4072e30 0x7f37f4165b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.551+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 >> v1:172.21.15.96:6789/0 conn(0x7f37f40b5230 legacy=0x7f37f4072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f40b4750 con 0x7f37f4158770 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f37f0001670 con 0x7f37f4158770 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 >> v1:172.21.15.70:6789/0 conn(0x7f37f4158770 legacy=0x7f37f415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc004c80 0x7f37dc005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37dc005910 0x7f37dc005d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37dc0062d0 0x7f37dc003440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37f40b4da0 con 0x7f37dc0062d0 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37dc003ab0 con 0x7f37dc005910 2024-04-15T15:16:00.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37dc003c60 con 0x7f37dc004c80 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37f4072e30 0x7f37f4165b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f8f59640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc004c80 0x7f37dc005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37dc0062d0 0x7f37dc003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37ebfff640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37dc005910 0x7f37dc005d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 -- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc004c80 msgr2=0x7f37dc005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc004c80 0x7f37dc005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 -- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37dc005910 msgr2=0x7f37dc005d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37dc005910 0x7f37dc005d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.552+0000 7f37f975a640 1 -- 172.21.15.70:0/955063226 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37dc004630 con 0x7f37dc0062d0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37f975a640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37dc0062d0 0x7f37dc003440 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f37f40d12d0 tx=0x7f37dc006940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec757f0f79ff6daa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec009d40 con 0x7f37dc0062d0 2024-04-15T15:16:00.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec02cce0 con 0x7f37dc0062d0 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37eb7fe640 1 -- 172.21.15.70:0/955063226 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec003bb0 con 0x7f37dc0062d0 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37fb1bd640 1 -- 172.21.15.70:0/955063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37dc0062d0 msgr2=0x7f37dc003440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/955063226 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37dc0062d0 0x7f37dc003440 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f37f40d12d0 tx=0x7f37dc006940 comp rx=0 tx=0).stop 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37f975a640 1 -- 172.21.15.70:0/955063226 reap_dead start 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37fb1bd640 1 -- 172.21.15.70:0/955063226 shutdown_connections 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.553+0000 7f37fb1bd640 1 -- 172.21.15.70:0/955063226 >> 172.21.15.70:0/955063226 conn(0x7f37f406be00 msgr2=0x7f37f406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.554+0000 7f37fb1bd640 1 -- 172.21.15.70:0/955063226 shutdown_connections 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.554+0000 7f37fb1bd640 1 -- 172.21.15.70:0/955063226 wait complete. 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.555+0000 7f37fb1bd640 1 Processor -- start 2024-04-15T15:16:00.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.555+0000 7f37fb1bd640 1 -- start start 2024-04-15T15:16:00.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 0x7f37f41f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 0x7f37f40cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4158b50 con 0x7f37f40ccca0 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f37f4071fa0 con 0x7f37f4147370 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f37f40b3d20 con 0x7f37dc0062d0 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43344/0 (socket says 172.21.15.70:43344) 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 -- 172.21.15.70:0/3315715065 learned_addr learned my addr 172.21.15.70:0/3315715065 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f8f59640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 0x7f37f41f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37ebfff640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 0x7f37f40cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 -- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 msgr2=0x7f37f41f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 0x7f37f41f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 -- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 msgr2=0x7f37f40cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 0x7f37f40cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 -- 172.21.15.70:0/3315715065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f41495a0 con 0x7f37f40ccca0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f8f59640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 0x7f37f41f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37ebfff640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 0x7f37f40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37f975a640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f37ec004260 tx=0x7f37ec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec009d40 con 0x7f37f40ccca0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37ec045070 con 0x7f37f40ccca0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f4149880 con 0x7f37f40ccca0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0045d0 con 0x7f37f40ccca0 2024-04-15T15:16:00.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.556+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f4149e40 con 0x7f37f40ccca0 2024-04-15T15:16:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.557+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37f40689e0 con 0x7f37f40ccca0 2024-04-15T15:16:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.558+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f37ec03d040 con 0x7f37f40ccca0 2024-04-15T15:16:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.558+0000 7f37e97fa640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 0x7f37e006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.558+0000 7f37f8f59640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 0x7f37e006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.558+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f37ec0ae6e0 con 0x7f37f40ccca0 2024-04-15T15:16:00.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.559+0000 7f37f8f59640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 0x7f37e006edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f37f0007c60 tx=0x7f37f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.563+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f37ec07c680 con 0x7f37f40ccca0 2024-04-15T15:16:00.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.751+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f37f4075350 con 0x7f37f40ccca0 2024-04-15T15:16:00.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.752+0000 7f37e97fa640 1 -- 172.21.15.70:0/3315715065 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f37ec07bf80 con 0x7f37f40ccca0 2024-04-15T15:16:00.754 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 msgr2=0x7f37e006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 0x7f37e006edb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f37f0007c60 tx=0x7f37f0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 msgr2=0x7f37f406eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f37ec004260 tx=0x7f37ec009750 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 shutdown_connections 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f37e006c8f0 0x7f37e006edb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f37f40ccca0 0x7f37f406eda0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f37f4147370 0x7f37f40cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 --2- 172.21.15.70:0/3315715065 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f37dc0062d0 0x7f37f41f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 >> 172.21.15.70:0/3315715065 conn(0x7f37f406be00 msgr2=0x7f37f40b0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 shutdown_connections 2024-04-15T15:16:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.754+0000 7f37fb1bd640 1 -- 172.21.15.70:0/3315715065 wait complete. 2024-04-15T15:16:00.767 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:00.767 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:00.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:00.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.889+0000 7fb33367e640 1 Processor -- start 2024-04-15T15:16:00.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.889+0000 7fb33367e640 1 -- start start 2024-04-15T15:16:00.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33367e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb32c0d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33367e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb32c0d4a40 0x7fb32c0d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33367e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb32c15eb70 con 0x7fb32c15e790 2024-04-15T15:16:00.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33367e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb32c0d3790 con 0x7fb32c0d4a40 2024-04-15T15:16:00.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33367e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb32c0d45b0 con 0x7fb32c0d15e0 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33141a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb32c15e790 0x7fb32c0b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57102/0 (socket says 172.21.15.70:57102) 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb33141a640 1 -- 172.21.15.70:0/2472805079 learned_addr learned my addr 172.21.15.70:0/2472805079 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2263559563 0 0) 0x7fb32c15eb70 con 0x7fb32c15e790 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb330c19640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb32c0d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2472805079 >> v2:172.21.15.96:3300/0 conn(0x7fb32c0d4a40 0x7fb32c0d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb318003520 con 0x7fb32c15e790 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3633878419 0 0) 0x7fb318003520 con 0x7fb32c15e790 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 msgr2=0x7fb32c0d1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb32c0d1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 >> v2:172.21.15.96:3300/0 conn(0x7fb32c0d4a40 msgr2=0x7fb32c0d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 --2- 172.21.15.70:0/2472805079 >> v2:172.21.15.96:3300/0 conn(0x7fb32c0d4a40 0x7fb32c0d4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c0d3c80 con 0x7fb32c15e790 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fb328001690 con 0x7fb32c15e790 2024-04-15T15:16:00.895 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 >> v1:172.21.15.70:6789/0 conn(0x7fb32c15e790 legacy=0x7fb32c0b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3180039b0 0x7fb318003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb318004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.890+0000 7fb323fff640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb318004d40 0x7fb3180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c0d3790 con 0x7fb318004d40 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c0d45b0 con 0x7fb3180039b0 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb31800ba70 con 0x7fb32c0d15e0 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb318004d40 0x7fb3180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb33141a640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3180039b0 0x7fb318003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb330c19640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb318004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 -- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 msgr2=0x7fb318004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c0d15e0 0x7fb318004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 -- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3180039b0 msgr2=0x7fb318003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb3180039b0 0x7fb318003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.896 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.891+0000 7fb331c1b640 1 -- 172.21.15.70:0/2472805079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb318003520 con 0x7fb318004d40 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb318004d40 0x7fb3180051c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb324009f90 tx=0x7fb318005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1294829a5dc324df server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324039070 con 0x7fb318004d40 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb32402cc20 con 0x7fb318004d40 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb323fff640 1 -- 172.21.15.70:0/2472805079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324009c10 con 0x7fb318004d40 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 -- 172.21.15.70:0/2472805079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb318004d40 msgr2=0x7fb3180051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 --2- 172.21.15.70:0/2472805079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb318004d40 0x7fb3180051c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb324009f90 tx=0x7fb318005830 comp rx=0 tx=0).stop 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb331c1b640 1 -- 172.21.15.70:0/2472805079 reap_dead start 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 -- 172.21.15.70:0/2472805079 shutdown_connections 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 -- 172.21.15.70:0/2472805079 >> 172.21.15.70:0/2472805079 conn(0x7fb32c067b40 msgr2=0x7fb32c067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 -- 172.21.15.70:0/2472805079 shutdown_connections 2024-04-15T15:16:00.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.892+0000 7fb33367e640 1 -- 172.21.15.70:0/2472805079 wait complete. 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 Processor -- start 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 -- start start 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 0x7fb32c1e9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 0x7fb32c0c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c0d4240 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb32c0d53f0 con 0x7fb318004d40 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb33367e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb32c0b7db0 con 0x7fb32c157ed0 2024-04-15T15:16:00.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb331c1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb331c1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43374/0 (socket says 172.21.15.70:43374) 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.894+0000 7fb331c1b640 1 -- 172.21.15.70:0/2478351478 learned_addr learned my addr 172.21.15.70:0/2478351478 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb330c19640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 0x7fb32c0c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb33141a640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 0x7fb32c1e9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 -- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 msgr2=0x7fb32c0c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 0x7fb32c0c9140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 -- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 msgr2=0x7fb32c1e9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 0x7fb32c1e9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 -- 172.21.15.70:0/2478351478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb32c0ca510 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb33141a640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 0x7fb32c1e9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb330c19640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 0x7fb32c0c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb331c1b640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb324037040 tx=0x7fb324009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324039030 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb32403a040 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb324003bb0 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb32c1f41f0 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.895+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb32c0c9c70 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.896+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb32c0689e0 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.897+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb324004030 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.897+0000 7fb321ffb640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 0x7fb31006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.897+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb3240ae170 con 0x7fb32c0cd0c0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.897+0000 7fb33141a640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 0x7fb31006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:00.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.898+0000 7fb33141a640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 0x7fb31006edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb328009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:00.903 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:00.902+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb32407bbe0 con 0x7fb32c0cd0c0 2024-04-15T15:16:01.078 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.077+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb32c0b6f80 con 0x7fb32c0cd0c0 2024-04-15T15:16:01.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.078+0000 7fb321ffb640 1 -- 172.21.15.70:0/2478351478 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fb32407fee0 con 0x7fb32c0cd0c0 2024-04-15T15:16:01.080 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:01.080 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 msgr2=0x7fb31006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 0x7fb31006edb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb328000c00 tx=0x7fb328009040 comp rx=0 tx=0).stop 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 msgr2=0x7fb32c0cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb324037040 tx=0x7fb324009750 comp rx=0 tx=0).stop 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 shutdown_connections 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb31006c8f0 0x7fb31006edb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb32c0cd0c0 0x7fb32c0cd540 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb32c157ed0 0x7fb32c0c9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 --2- 172.21.15.70:0/2478351478 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb318004d40 0x7fb32c1e9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 >> 172.21.15.70:0/2478351478 conn(0x7fb32c067b40 msgr2=0x7fb32c0b8820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 shutdown_connections 2024-04-15T15:16:01.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.080+0000 7fb33367e640 1 -- 172.21.15.70:0/2478351478 wait complete. 2024-04-15T15:16:01.093 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:01.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:01.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.199+0000 7f186aeba640 1 Processor -- start 2024-04-15T15:16:01.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.199+0000 7f186aeba640 1 -- start start 2024-04-15T15:16:01.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f186aeba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f186415e830 0x7f18640b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f186aeba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f186415ec10 0x7f1864067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f186aeba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18640b3c10 con 0x7f1864067c60 2024-04-15T15:16:01.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f186aeba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f18640b3ef0 con 0x7f186415e830 2024-04-15T15:16:01.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f186aeba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f18640b41d0 con 0x7f186415ec10 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f186415e830 0x7f18640b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f186415e830 0x7f18640b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43386/0 (socket says 172.21.15.70:43386) 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 -- 172.21.15.70:0/1028001325 learned_addr learned my addr 172.21.15.70:0/1028001325 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f185bfff640 1 --2- 172.21.15.70:0/1028001325 >> v2:172.21.15.96:3300/0 conn(0x7f186415ec10 0x7f1864067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 -- 172.21.15.70:0/1028001325 >> v2:172.21.15.96:3300/0 conn(0x7f186415ec10 msgr2=0x7f1864067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 --2- 172.21.15.70:0/1028001325 >> v2:172.21.15.96:3300/0 conn(0x7f186415ec10 0x7f1864067720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 -- 172.21.15.70:0/1028001325 >> v1:172.21.15.96:6789/0 conn(0x7f1864067c60 legacy=0x7f18640b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.203 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.200+0000 7f1868c56640 1 -- 172.21.15.70:0/1028001325 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18640b44b0 con 0x7f186415e830 2024-04-15T15:16:01.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.201+0000 7f1868c56640 1 --2- 172.21.15.70:0/1028001325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f186415e830 0x7f18640b4cb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f186406e050 tx=0x7f186002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbe1b49f4ee3762 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.201+0000 7f185b7fe640 1 -- 172.21.15.70:0/1028001325 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186002ca80 con 0x7f186415e830 2024-04-15T15:16:01.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.201+0000 7f185b7fe640 1 -- 172.21.15.70:0/1028001325 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f186002cc20 con 0x7f186415e830 2024-04-15T15:16:01.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.201+0000 7f185b7fe640 1 -- 172.21.15.70:0/1028001325 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18600343f0 con 0x7f186415e830 2024-04-15T15:16:01.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 -- 172.21.15.70:0/1028001325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f186415e830 msgr2=0x7f18640b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 --2- 172.21.15.70:0/1028001325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f186415e830 0x7f18640b4cb0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f186406e050 tx=0x7f186002bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 -- 172.21.15.70:0/1028001325 shutdown_connections 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 --2- 172.21.15.70:0/1028001325 >> v2:172.21.15.96:3300/0 conn(0x7f186415ec10 0x7f1864067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 --2- 172.21.15.70:0/1028001325 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f186415e830 0x7f18640b4cb0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 -- 172.21.15.70:0/1028001325 >> 172.21.15.70:0/1028001325 conn(0x7f1864073360 msgr2=0x7f186406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 -- 172.21.15.70:0/1028001325 shutdown_connections 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.202+0000 7f186aeba640 1 -- 172.21.15.70:0/1028001325 wait complete. 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.203+0000 7f186aeba640 1 Processor -- start 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.203+0000 7f186aeba640 1 -- start start 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1864067c60 0x7f1864147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f186415e830 0x7f18640c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18641600e0 con 0x7f18640c6020 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f186415ff60 con 0x7f1864067c60 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f186aeba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f18640b5ad0 con 0x7f186415e830 2024-04-15T15:16:01.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43398/0 (socket says 172.21.15.70:43398) 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 -- 172.21.15.70:0/3945985842 learned_addr learned my addr 172.21.15.70:0/3945985842 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f185bfff640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f186415e830 0x7f18640c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1868c56640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1864067c60 0x7f1864147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 -- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f186415e830 msgr2=0x7f18640c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f186415e830 0x7f18640c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 -- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1864067c60 msgr2=0x7f1864147b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1864067c60 0x7f1864147b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 -- 172.21.15.70:0/3945985842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641e7860 con 0x7f18640c6020 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.204+0000 7f1869457640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f185c00c400 tx=0x7f185c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.205+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c017070 con 0x7f18640c6020 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.205+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f185c0052d0 con 0x7f18640c6020 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.205+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1864148550 con 0x7f18640c6020 2024-04-15T15:16:01.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.205+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185c003810 con 0x7f18640c6020 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.205+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1864148070 con 0x7f18640c6020 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.206+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18640b6540 con 0x7f18640c6020 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.206+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f185c01a070 con 0x7f18640c6020 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.206+0000 7f18597fa640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 0x7f185006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.207+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f185c08b2c0 con 0x7f18640c6020 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.207+0000 7f1868c56640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 0x7f185006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.207+0000 7f1868c56640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 0x7f185006edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1860035000 tx=0x7f1860030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.211+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f18640b6540 con 0x7f18640c6020 2024-04-15T15:16:01.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.388+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18640698a0 con 0x7f18640c6020 2024-04-15T15:16:01.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.388+0000 7f18597fa640 1 -- 172.21.15.70:0/3945985842 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f18640698a0 con 0x7f18640c6020 2024-04-15T15:16:01.390 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:01.392 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.390+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 msgr2=0x7f185006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.390+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 0x7f185006edb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1860035000 tx=0x7f1860030000 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.390+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 msgr2=0x7f18641e6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.390+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f185c00c400 tx=0x7f185c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 shutdown_connections 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f185006c8f0 0x7f185006edb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f18640c6020 0x7f18641e6b70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f186415e830 0x7f18640c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 --2- 172.21.15.70:0/3945985842 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1864067c60 0x7f1864147b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 >> 172.21.15.70:0/3945985842 conn(0x7f1864073360 msgr2=0x7f18640b0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 shutdown_connections 2024-04-15T15:16:01.393 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.391+0000 7f186aeba640 1 -- 172.21.15.70:0/3945985842 wait complete. 2024-04-15T15:16:01.404 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.508+0000 7fd9db5a8640 1 Processor -- start 2024-04-15T15:16:01.510 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.508+0000 7fd9db5a8640 1 -- start start 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9db5a8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9db5a8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9db5a8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9d40b10d0 con 0x7fd9d40b4350 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9db5a8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9d415f090 con 0x7fd9d415ecb0 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9db5a8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d40cdf80 con 0x7fd9d40b1d30 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d9344640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d9b45640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d9344640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:58490/0 (socket says 172.21.15.70:58490) 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d9b45640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:53164/0 (socket says 172.21.15.70:53164) 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d8b43640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd9d40b4350 0x7fd9d40b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:60878/0 (socket says 172.21.15.70:60878) 2024-04-15T15:16:01.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.509+0000 7fd9d9b45640 1 -- 172.21.15.70:0/1266071151 learned_addr learned my addr 172.21.15.70:0/1266071151 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9b45640 1 -- 172.21.15.70:0/1266071151 >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 msgr2=0x7fd9d40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9b45640 1 --2- 172.21.15.70:0/1266071151 >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9b45640 1 -- 172.21.15.70:0/1266071151 >> v1:172.21.15.96:6789/0 conn(0x7fd9d40b4350 legacy=0x7fd9d40b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9b45640 1 -- 172.21.15.70:0/1266071151 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d40af920 con 0x7fd9d40b1d30 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9cbfff640 1 -- 172.21.15.70:0/1266071151 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048147316 0 0) 0x7fd9d40b10d0 con 0x7fd9d40b4350 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9344640 1 --2- 172.21.15.70:0/1266071151 >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.510+0000 7fd9d9b45640 1 --2- 172.21.15.70:0/1266071151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009e50 tx=0x7fd9cc02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409d70893e19eb7a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9cbfff640 1 -- 172.21.15.70:0/1266071151 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc02cb90 con 0x7fd9d40b1d30 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9cbfff640 1 -- 172.21.15.70:0/1266071151 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9cc02cd30 con 0x7fd9d40b1d30 2024-04-15T15:16:01.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9cbfff640 1 -- 172.21.15.70:0/1266071151 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc034610 con 0x7fd9d40b1d30 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 -- 172.21.15.70:0/1266071151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 msgr2=0x7fd9d40af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/1266071151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009e50 tx=0x7fd9cc02bff0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 -- 172.21.15.70:0/1266071151 shutdown_connections 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/1266071151 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d40af340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/1266071151 >> v2:172.21.15.96:3300/0 conn(0x7fd9d415ecb0 0x7fd9d40b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 -- 172.21.15.70:0/1266071151 >> 172.21.15.70:0/1266071151 conn(0x7fd9d40b6870 msgr2=0x7fd9d40b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 -- 172.21.15.70:0/1266071151 shutdown_connections 2024-04-15T15:16:01.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.511+0000 7fd9db5a8640 1 -- 172.21.15.70:0/1266071151 wait complete. 2024-04-15T15:16:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.512+0000 7fd9db5a8640 1 Processor -- start 2024-04-15T15:16:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.512+0000 7fd9db5a8640 1 -- start start 2024-04-15T15:16:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d4153f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 0x7fd9d415fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d40b47c0 con 0x7fd9d40b4350 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9d40afee0 con 0x7fd9d415ecb0 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9db5a8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd9d41544c0 con 0x7fd9d40b1d30 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9d8b43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9d8b43640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43408/0 (socket says 172.21.15.70:43408) 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9d8b43640 1 -- 172.21.15.70:0/3616264356 learned_addr learned my addr 172.21.15.70:0/3616264356 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.513+0000 7fd9d9344640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d4153f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d9b45640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 0x7fd9d415fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 -- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 msgr2=0x7fd9d4153f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d4153f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 -- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 msgr2=0x7fd9d415fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 0x7fd9d415fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 -- 172.21.15.70:0/3616264356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9d4158290 con 0x7fd9d40b4350 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d9b45640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 0x7fd9d415fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:01.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9d8b43640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c4a0 tx=0x7fd9c400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4018070 con 0x7fd9d40b4350 2024-04-15T15:16:01.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c40074e0 con 0x7fd9d40b4350 2024-04-15T15:16:01.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9d4158570 con 0x7fd9d40b4350 2024-04-15T15:16:01.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004a20 con 0x7fd9d40b4350 2024-04-15T15:16:01.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.514+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9d4158b30 con 0x7fd9d40b4350 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.515+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9d40689e0 con 0x7fd9d40b4350 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.515+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9c401b070 con 0x7fd9d40b4350 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.516+0000 7fd9c9ffb640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 0x7fd9b806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.516+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd9c408b200 con 0x7fd9d40b4350 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.516+0000 7fd9d9344640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 0x7fd9b806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.517+0000 7fd9d9344640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 0x7fd9b806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009560 tx=0x7fd9d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.520+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9c4058be0 con 0x7fd9d40b4350 2024-04-15T15:16:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.694+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd9d406c7c0 con 0x7fd9d40b4350 2024-04-15T15:16:01.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.694+0000 7fd9c9ffb640 1 -- 172.21.15.70:0/3616264356 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd9c405cee0 con 0x7fd9d40b4350 2024-04-15T15:16:01.696 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:01.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 msgr2=0x7fd9b806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 0x7fd9b806edb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd9d0009560 tx=0x7fd9d0009190 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 msgr2=0x7fd9d4161890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c4a0 tx=0x7fd9c400c960 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 shutdown_connections 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9d415ecb0 0x7fd9d415fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9b806c8f0 0x7fd9b806edb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9d40b4350 0x7fd9d4161890 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 --2- 172.21.15.70:0/3616264356 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9d40b1d30 0x7fd9d4153f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 >> 172.21.15.70:0/3616264356 conn(0x7fd9d40b6870 msgr2=0x7fd9d40b4ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 shutdown_connections 2024-04-15T15:16:01.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.696+0000 7fd9db5a8640 1 -- 172.21.15.70:0/3616264356 wait complete. 2024-04-15T15:16:01.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.813+0000 7f79b6332640 1 Processor -- start 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.813+0000 7f79b6332640 1 -- start start 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b6332640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f79b0071d10 0x7f79b00d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b6332640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b015db40 con 0x7f79b00d0bd0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b6332640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79b0072170 con 0x7f79b0072d80 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b6332640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f79b00d1040 con 0x7f79b0071d10 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f79b0071d10 0x7f79b00d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b48cf640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f79b00d0bd0 0x7f79b006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57104/0 (socket says 172.21.15.70:57104) 2024-04-15T15:16:01.821 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79af7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f79b0071d10 0x7f79b00d0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43424/0 (socket says 172.21.15.70:43424) 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79b48cf640 1 -- 172.21.15.70:0/3237059949 learned_addr learned my addr 172.21.15.70:0/3237059949 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79af7fe640 1 -- 172.21.15.70:0/3237059949 >> v1:172.21.15.96:6789/0 conn(0x7f79b0072d80 legacy=0x7f79b00af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79af7fe640 1 -- 172.21.15.70:0/3237059949 >> v1:172.21.15.70:6789/0 conn(0x7f79b00d0bd0 legacy=0x7f79b006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.814+0000 7f79af7fe640 1 -- 172.21.15.70:0/3237059949 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b015cfc0 con 0x7f79b0071d10 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.815+0000 7f79af7fe640 1 --2- 172.21.15.70:0/3237059949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b00d0690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79a40097a0 tx=0x7f79a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556b3ba49085c944 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.815+0000 7f79aeffd640 1 -- 172.21.15.70:0/3237059949 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4035020 con 0x7f79b0071d10 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.815+0000 7f79aeffd640 1 -- 172.21.15.70:0/3237059949 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a4031830 con 0x7f79b0071d10 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.815+0000 7f79aeffd640 1 -- 172.21.15.70:0/3237059949 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a4031c00 con 0x7f79b0071d10 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 -- 172.21.15.70:0/3237059949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 msgr2=0x7f79b00d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 --2- 172.21.15.70:0/3237059949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b00d0690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f79a40097a0 tx=0x7f79a402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 -- 172.21.15.70:0/3237059949 shutdown_connections 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 --2- 172.21.15.70:0/3237059949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b00d0690 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 -- 172.21.15.70:0/3237059949 >> 172.21.15.70:0/3237059949 conn(0x7f79b006be00 msgr2=0x7f79b006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 -- 172.21.15.70:0/3237059949 shutdown_connections 2024-04-15T15:16:01.822 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.816+0000 7f79b6332640 1 -- 172.21.15.70:0/3237059949 wait complete. 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.817+0000 7f79b6332640 1 Processor -- start 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.817+0000 7f79b6332640 1 -- start start 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 0x7f79b015acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 0x7f79b015b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f79b015d860 con 0x7f79b0071d10 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f79b015d580 con 0x7f79b0072d80 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b6332640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f79b00afaf0 con 0x7f79b00d0bd0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43430/0 (socket says 172.21.15.70:43430) 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 -- 172.21.15.70:0/992066284 learned_addr learned my addr 172.21.15.70:0/992066284 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79af7fe640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 0x7f79b015acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b48cf640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 0x7f79b015b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 -- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 msgr2=0x7f79b015b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 0x7f79b015b200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 -- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 msgr2=0x7f79b015acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 0x7f79b015acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 -- 172.21.15.70:0/992066284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79b00cf2d0 con 0x7f79b0071d10 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79af7fe640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 0x7f79b015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79b48cf640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 0x7f79b015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.818+0000 7f79affff640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c400 tx=0x7f79a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.819+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a00052d0 con 0x7f79b0071d10 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.819+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79a0005490 con 0x7f79b0071d10 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.819+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79b00cf5b0 con 0x7f79b0071d10 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.819+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79a0007260 con 0x7f79b0071d10 2024-04-15T15:16:01.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.819+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79b009a2f0 con 0x7f79b0071d10 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.820+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79b00689e0 con 0x7f79b0071d10 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.820+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f79a0007400 con 0x7f79b0071d10 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.820+0000 7f79acff9640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 0x7f799c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.821+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f79a008b4e0 con 0x7f79b0071d10 2024-04-15T15:16:01.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.821+0000 7f79af7fe640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 0x7f799c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:01.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.821+0000 7f79af7fe640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 0x7f799c06edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f79a4036000 tx=0x7f79a4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:01.828 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.825+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f79a0059510 con 0x7f79b0071d10 2024-04-15T15:16:02.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:01.999+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f79b00b2c00 con 0x7f79b0071d10 2024-04-15T15:16:02.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.000+0000 7f79acff9640 1 -- 172.21.15.70:0/992066284 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f79a0058e10 con 0x7f79b0071d10 2024-04-15T15:16:02.002 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:02.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 msgr2=0x7f799c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 0x7f799c06edb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f79a4036000 tx=0x7f79a4008030 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 msgr2=0x7f79b0149730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f79a000c400 tx=0x7f79a000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 shutdown_connections 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f79b00d0bd0 0x7f79b015b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f79b0072d80 0x7f79b015acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f799c06c8f0 0x7f799c06edb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 --2- 172.21.15.70:0/992066284 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f79b0071d10 0x7f79b0149730 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 >> 172.21.15.70:0/992066284 conn(0x7f79b006be00 msgr2=0x7f79b0074cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 shutdown_connections 2024-04-15T15:16:02.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.002+0000 7f79b6332640 1 -- 172.21.15.70:0/992066284 wait complete. 2024-04-15T15:16:02.015 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.120+0000 7f552d60d640 1 Processor -- start 2024-04-15T15:16:02.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.120+0000 7f552d60d640 1 -- start start 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f552d60d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55280d15e0 0x7f55280d1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f552d60d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55280d4a40 0x7f55280d4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f552d60d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f552815eb70 con 0x7f552815e790 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f552d60d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f55280d3790 con 0x7f55280d15e0 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f552d60d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f55280d45b0 con 0x7f55280d4a40 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f551e7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55280d15e0 0x7f55280d1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f5526ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f552815e790 0x7f55280b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:57120/0 (socket says 172.21.15.70:57120) 2024-04-15T15:16:02.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f5526ffd640 1 -- 172.21.15.70:0/4068719889 learned_addr learned my addr 172.21.15.70:0/4068719889 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f55277fe640 1 --2- 172.21.15.70:0/4068719889 >> v2:172.21.15.96:3300/0 conn(0x7f55280d4a40 0x7f55280d4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f551e7fc640 1 -- 172.21.15.70:0/4068719889 >> v2:172.21.15.96:3300/0 conn(0x7f55280d4a40 msgr2=0x7f55280d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f551e7fc640 1 --2- 172.21.15.70:0/4068719889 >> v2:172.21.15.96:3300/0 conn(0x7f55280d4a40 0x7f55280d4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f551e7fc640 1 -- 172.21.15.70:0/4068719889 >> v1:172.21.15.70:6789/0 conn(0x7f552815e790 legacy=0x7f55280b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f551e7fc640 1 -- 172.21.15.70:0/4068719889 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55280d3c80 con 0x7f55280d15e0 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.121+0000 7f55267fc640 1 -- 172.21.15.70:0/4068719889 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317595198 0 0) 0x7f552815eb70 con 0x7f552815e790 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.122+0000 7f551e7fc640 1 --2- 172.21.15.70:0/4068719889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55280d15e0 0x7f55280d1a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55280b61c0 tx=0x7f551002b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=619e79d55028e2f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.122+0000 7f55267fc640 1 -- 172.21.15.70:0/4068719889 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510038070 con 0x7f55280d15e0 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.122+0000 7f55267fc640 1 -- 172.21.15.70:0/4068719889 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5510031710 con 0x7f55280d15e0 2024-04-15T15:16:02.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.122+0000 7f55267fc640 1 -- 172.21.15.70:0/4068719889 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5510031ae0 con 0x7f55280d15e0 2024-04-15T15:16:02.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 -- 172.21.15.70:0/4068719889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55280d15e0 msgr2=0x7f55280d1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 --2- 172.21.15.70:0/4068719889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55280d15e0 0x7f55280d1a90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55280b61c0 tx=0x7f551002b7b0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 -- 172.21.15.70:0/4068719889 shutdown_connections 2024-04-15T15:16:02.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 --2- 172.21.15.70:0/4068719889 >> v2:172.21.15.96:3300/0 conn(0x7f55280d4a40 0x7f55280d4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 --2- 172.21.15.70:0/4068719889 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55280d15e0 0x7f55280d1a90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 -- 172.21.15.70:0/4068719889 >> 172.21.15.70:0/4068719889 conn(0x7f5528067b40 msgr2=0x7f5528067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 -- 172.21.15.70:0/4068719889 shutdown_connections 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.123+0000 7f552d60d640 1 -- 172.21.15.70:0/4068719889 wait complete. 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.124+0000 7f552d60d640 1 Processor -- start 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.124+0000 7f552d60d640 1 -- start start 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55280d15e0 0x7f552814a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55280d4a40 0x7f55280cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55280d4240 con 0x7f552815e790 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55280d53f0 con 0x7f55280d4a40 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f552d60d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f55280b7db0 con 0x7f55280d15e0 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:43448/0 (socket says 172.21.15.70:43448) 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 -- 172.21.15.70:0/2968614533 learned_addr learned my addr 172.21.15.70:0/2968614533 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:02.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f5526ffd640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55280d15e0 0x7f552814a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 -- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55280d15e0 msgr2=0x7f552814a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f551e7fc640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55280d4a40 0x7f55280cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55280d15e0 0x7f552814a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 -- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55280d4a40 msgr2=0x7f55280cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55280d4a40 0x7f55280cb6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.125+0000 7f55277fe640 1 -- 172.21.15.70:0/2968614533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55280cc180 con 0x7f552815e790 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f55277fe640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5520007cf0 tx=0x7f552000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520014420 con 0x7f552815e790 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5520019070 con 0x7f552815e790 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55280cc460 con 0x7f552815e790 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5520014820 con 0x7f552815e790 2024-04-15T15:16:02.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.126+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55280cca20 con 0x7f552815e790 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.127+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5528073c40 con 0x7f552815e790 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.127+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55200149c0 con 0x7f552815e790 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.128+0000 7f551ffff640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 0x7f550806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.128+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f552008c730 con 0x7f552815e790 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.128+0000 7f5526ffd640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 0x7f550806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.128+0000 7f5526ffd640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 0x7f550806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5518000f60 tx=0x7f5518009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.132+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5528073c40 con 0x7f552815e790 2024-04-15T15:16:02.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.307+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f55280689e0 con 0x7f552815e790 2024-04-15T15:16:02.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.307+0000 7f551ffff640 1 -- 172.21.15.70:0/2968614533 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f552005a760 con 0x7f552815e790 2024-04-15T15:16:02.309 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:02.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 msgr2=0x7f550806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 0x7f550806edb0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5518000f60 tx=0x7f5518009040 comp rx=0 tx=0).stop 2024-04-15T15:16:02.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 msgr2=0x7f55280cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5520007cf0 tx=0x7f552000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 shutdown_connections 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f550806c8f0 0x7f550806edb0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f552815e790 0x7f55280cd8f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55280d4a40 0x7f55280cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 --2- 172.21.15.70:0/2968614533 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55280d15e0 0x7f552814a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 >> 172.21.15.70:0/2968614533 conn(0x7f5528067b40 msgr2=0x7f55280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 shutdown_connections 2024-04-15T15:16:02.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.309+0000 7f552d60d640 1 -- 172.21.15.70:0/2968614533 wait complete. 2024-04-15T15:16:02.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:02.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.425+0000 7fc5725f4640 1 Processor -- start 2024-04-15T15:16:02.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.425+0000 7fc5725f4640 1 -- start start 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc5725f4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc56c071d10 0x7fc56c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc5725f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc56c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc5725f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c15db40 con 0x7fc56c072d80 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc5725f4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc56c072170 con 0x7fc56c071d10 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc5725f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c15cfc0 con 0x7fc56c0d0bd0 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc56c072d80 0x7fc56c0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51308/0 (socket says 172.21.15.70:51308) 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56bfff640 1 -- 172.21.15.70:0/3987535953 learned_addr learned my addr 172.21.15.70:0/3987535953 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3659287432 0 0) 0x7fc56c15db40 con 0x7fc56c072d80 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc570b91640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc56c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc554003520 con 0x7fc56c072d80 2024-04-15T15:16:02.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3987535953 >> v2:172.21.15.96:3300/0 conn(0x7fc56c071d10 0x7fc56c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 424658637 0 0) 0x7fc554003520 con 0x7fc56c072d80 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 msgr2=0x7fc56c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc56c06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 >> v2:172.21.15.96:3300/0 conn(0x7fc56c071d10 msgr2=0x7fc56c0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 --2- 172.21.15.70:0/3987535953 >> v2:172.21.15.96:3300/0 conn(0x7fc56c071d10 0x7fc56c0d0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c15d2a0 con 0x7fc56c072d80 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fc55c001670 con 0x7fc56c072d80 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 >> v1:172.21.15.70:6789/0 conn(0x7fc56c072d80 legacy=0x7fc56c0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5540039b0 0x7fc554003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc554004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc554004d40 0x7fc5540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c072170 con 0x7fc5540039b0 2024-04-15T15:16:02.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c15cfc0 con 0x7fc554004d40 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc55400b9c0 con 0x7fc56c0d0bd0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.426+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5540039b0 0x7fc554003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc554004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc570b91640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc554004d40 0x7fc5540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 -- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 msgr2=0x7fc554004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc554004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 -- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc554004d40 msgr2=0x7fc5540051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc554004d40 0x7fc5540051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 -- 172.21.15.70:0/3987535953 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc554003520 con 0x7fc5540039b0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc56c0d0bd0 0x7fc554004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc570b91640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc554004d40 0x7fc5540051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.427+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5540039b0 0x7fc554003dd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001050 tx=0x7fc554005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8babd7554f163ad server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c007d70 con 0x7fc5540039b0 2024-04-15T15:16:02.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc55c030070 con 0x7fc5540039b0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc56affd640 1 -- 172.21.15.70:0/3987535953 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc55c02cd80 con 0x7fc5540039b0 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 -- 172.21.15.70:0/3987535953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5540039b0 msgr2=0x7fc554003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3987535953 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5540039b0 0x7fc554003dd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc55c001050 tx=0x7fc554005830 comp rx=0 tx=0).stop 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc570b91640 1 -- 172.21.15.70:0/3987535953 reap_dead start 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 -- 172.21.15.70:0/3987535953 shutdown_connections 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 -- 172.21.15.70:0/3987535953 >> 172.21.15.70:0/3987535953 conn(0x7fc56c06be00 msgr2=0x7fc56c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 -- 172.21.15.70:0/3987535953 shutdown_connections 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.428+0000 7fc5725f4640 1 -- 172.21.15.70:0/3987535953 wait complete. 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.429+0000 7fc5725f4640 1 Processor -- start 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 -- start start 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 0x7fc56c1f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 0x7fc56c0afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c0afaf0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c15d860 con 0x7fc56c1f5180 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc5725f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c15cb90 con 0x7fc5540039b0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc56b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc56b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41522/0 (socket says 172.21.15.70:41522) 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.430+0000 7fc56b7fe640 1 -- 172.21.15.70:0/3143014131 learned_addr learned my addr 172.21.15.70:0/3143014131 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 0x7fc56c1f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 -- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 msgr2=0x7fc56c1f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc570b91640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 0x7fc56c0afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 0x7fc56c1f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 -- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 msgr2=0x7fc56c0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 0x7fc56c0afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 -- 172.21.15.70:0/3143014131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0ce0e0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 0x7fc56c1f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc570b91640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 0x7fc56c0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc56b7fe640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc5600116d0 tx=0x7fc560011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc56000f9c0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56000fb60 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c0ce3c0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc560015460 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.431+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c0ce980 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.432+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0689e0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.432+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc560015600 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.433+0000 7fc568ff9640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 0x7fc55806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.433+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 0x7fc55806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.433+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc56008c790 con 0x7fc56c1f4d00 2024-04-15T15:16:02.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.434+0000 7fc56bfff640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 0x7fc55806ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:02.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.437+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc560019030 con 0x7fc56c1f4d00 2024-04-15T15:16:02.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.627+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc56c0690f0 con 0x7fc56c1f4d00 2024-04-15T15:16:02.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.628+0000 7fc568ff9640 1 -- 172.21.15.70:0/3143014131 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fc56005a850 con 0x7fc56c1f4d00 2024-04-15T15:16:02.629 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 msgr2=0x7fc55806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 0x7fc55806ede0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc55c034000 tx=0x7fc55c007000 comp rx=0 tx=0).stop 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 msgr2=0x7fc56c0cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc5600116d0 tx=0x7fc560011b90 comp rx=0 tx=0).stop 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 shutdown_connections 2024-04-15T15:16:02.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc56c1f5180 0x7fc56c0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc55806c920 0x7fc55806ede0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc56c1f4d00 0x7fc56c0cd690 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 --2- 172.21.15.70:0/3143014131 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5540039b0 0x7fc56c1f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.629+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 >> 172.21.15.70:0/3143014131 conn(0x7fc56c06be00 msgr2=0x7fc56c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.630+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 shutdown_connections 2024-04-15T15:16:02.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:02.630+0000 7fc5725f4640 1 -- 172.21.15.70:0/3143014131 wait complete. 2024-04-15T15:16:02.643 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-15T15:16:02.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:02.643 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:02.678 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:02.678 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:16:02.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:02.678 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:16:02.812 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:02.812 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:02.828 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:02.828 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:02.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:03.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.025+0000 7fdb58711640 1 Processor -- start 2024-04-15T15:16:03.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.025+0000 7fdb58711640 1 -- start start 2024-04-15T15:16:03.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb58711640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb5015ecb0 0x7fdb500b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb58711640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 0x7fdb500af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb58711640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb500b10d0 con 0x7fdb500b4350 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb58711640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdb5015f090 con 0x7fdb5015ecb0 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb58711640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fdb500cdf80 con 0x7fdb500b1d30 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb5015ecb0 0x7fdb500b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb55cac640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdb500b4350 0x7fdb500b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51314/0 (socket says 172.21.15.70:51314) 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdb5015ecb0 0x7fdb500b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41526/0 (socket says 172.21.15.70:41526) 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb55cac640 1 -- 172.21.15.70:0/2902163164 learned_addr learned my addr 172.21.15.70:0/2902163164 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb56cae640 1 --2- 172.21.15.70:0/2902163164 >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 0x7fdb500af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 -- 172.21.15.70:0/2902163164 >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 msgr2=0x7fdb500af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 --2- 172.21.15.70:0/2902163164 >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 0x7fdb500af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 -- 172.21.15.70:0/2902163164 >> v1:172.21.15.70:6789/0 conn(0x7fdb500b4350 legacy=0x7fdb500b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb564ad640 1 -- 172.21.15.70:0/2902163164 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb500af920 con 0x7fdb5015ecb0 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb554ab640 1 -- 172.21.15.70:0/2902163164 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451000330 0 0) 0x7fdb500b10d0 con 0x7fdb500b4350 2024-04-15T15:16:03.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.026+0000 7fdb56cae640 1 --2- 172.21.15.70:0/2902163164 >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 0x7fdb500af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.027+0000 7fdb564ad640 1 --2- 172.21.15.70:0/2902163164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb5015ecb0 0x7fdb500b0b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0097a0 tx=0x7fdb4c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2700e99bbdcec4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.027+0000 7fdb554ab640 1 -- 172.21.15.70:0/2902163164 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c034070 con 0x7fdb5015ecb0 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.027+0000 7fdb554ab640 1 -- 172.21.15.70:0/2902163164 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb4c030610 con 0x7fdb5015ecb0 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.027+0000 7fdb554ab640 1 -- 172.21.15.70:0/2902163164 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c0309e0 con 0x7fdb5015ecb0 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 -- 172.21.15.70:0/2902163164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb5015ecb0 msgr2=0x7fdb500b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 --2- 172.21.15.70:0/2902163164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb5015ecb0 0x7fdb500b0b90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c0097a0 tx=0x7fdb4c02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 -- 172.21.15.70:0/2902163164 shutdown_connections 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 --2- 172.21.15.70:0/2902163164 >> v2:172.21.15.96:3300/0 conn(0x7fdb500b1d30 0x7fdb500af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 --2- 172.21.15.70:0/2902163164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb5015ecb0 0x7fdb500b0b90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 -- 172.21.15.70:0/2902163164 >> 172.21.15.70:0/2902163164 conn(0x7fdb500b6870 msgr2=0x7fdb500b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 -- 172.21.15.70:0/2902163164 shutdown_connections 2024-04-15T15:16:03.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.028+0000 7fdb58711640 1 -- 172.21.15.70:0/2902163164 wait complete. 2024-04-15T15:16:03.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.029+0000 7fdb58711640 1 Processor -- start 2024-04-15T15:16:03.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.029+0000 7fdb58711640 1 -- start start 2024-04-15T15:16:03.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb500b1d30 0x7fdb50153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb5015ecb0 0x7fdb500d0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb500b47c0 con 0x7fdb500b4350 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb500afee0 con 0x7fdb5015ecb0 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb58711640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdb500ca6d0 con 0x7fdb500b1d30 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41530/0 (socket says 172.21.15.70:41530) 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 -- 172.21.15.70:0/612287342 learned_addr learned my addr 172.21.15.70:0/612287342 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb564ad640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb500b1d30 0x7fdb50153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 -- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb500b1d30 msgr2=0x7fdb50153c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb500b1d30 0x7fdb50153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 -- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb5015ecb0 msgr2=0x7fdb500d0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb5015ecb0 0x7fdb500d0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 -- 172.21.15.70:0/612287342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb50154be0 con 0x7fdb500b4350 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.030+0000 7fdb55cac640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c4d0 tx=0x7fdb4000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.031+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40018070 con 0x7fdb500b4350 2024-04-15T15:16:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.031+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb40007500 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.031+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb5015ffb0 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.031+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb40004a20 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.031+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb500cac00 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.032+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb500689e0 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.032+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb4001b070 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.032+0000 7fdb46ffd640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 0x7fdb3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.033+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdb4008b1b0 con 0x7fdb500b4350 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.033+0000 7fdb564ad640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 0x7fdb3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.033+0000 7fdb564ad640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 0x7fdb3406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c036000 tx=0x7fdb4c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.037+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb50154be0 con 0x7fdb500b4350 2024-04-15T15:16:03.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.213+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdb5006c7c0 con 0x7fdb500b4350 2024-04-15T15:16:03.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.214+0000 7fdb46ffd640 1 -- 172.21.15.70:0/612287342 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fdb40059270 con 0x7fdb500b4350 2024-04-15T15:16:03.216 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:03.216 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 msgr2=0x7fdb3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 0x7fdb3406ede0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c036000 tx=0x7fdb4c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 msgr2=0x7fdb500d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdb4000c4d0 tx=0x7fdb4000c990 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 shutdown_connections 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb5015ecb0 0x7fdb500d0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdb3406c920 0x7fdb3406ede0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb500b4350 0x7fdb500d0900 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 --2- 172.21.15.70:0/612287342 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb500b1d30 0x7fdb50153c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 >> 172.21.15.70:0/612287342 conn(0x7fdb500b6870 msgr2=0x7fdb500b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 shutdown_connections 2024-04-15T15:16:03.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.216+0000 7fdb58711640 1 -- 172.21.15.70:0/612287342 wait complete. 2024-04-15T15:16:03.229 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:03.230 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:03.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.334+0000 7f368a6c5640 1 Processor -- start 2024-04-15T15:16:03.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.334+0000 7f368a6c5640 1 -- start start 2024-04-15T15:16:03.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f368a6c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f36840b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f368a6c5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36840b1f40 con 0x7f368415fda0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f368a6c5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36840cda30 con 0x7f36840b1480 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f368a6c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f36840cdd10 con 0x7f36840cff10 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f3683fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f368415fda0 0x7f36840ce7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51324/0 (socket says 172.21.15.70:51324) 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f3683fff640 1 -- 172.21.15.70:0/2585471735 learned_addr learned my addr 172.21.15.70:0/2585471735 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064103930 0 0) 0x7f36840b1f40 con 0x7f368415fda0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f3688c62640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f36840b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.335+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f366c003520 con 0x7f368415fda0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3488859581 0 0) 0x7f366c003520 con 0x7f368415fda0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 msgr2=0x7f36840b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f36840b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 >> v1:172.21.15.96:6789/0 conn(0x7f36840b1480 legacy=0x7f36840cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36840cdff0 con 0x7f368415fda0 2024-04-15T15:16:03.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f3674001670 con 0x7f368415fda0 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 >> v1:172.21.15.70:6789/0 conn(0x7f368415fda0 legacy=0x7f36840ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f366c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f366c003f60 0x7f366c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f366c004d40 0x7f366c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36840cdd10 con 0x7f366c004d40 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f366c00baa0 con 0x7f366c003f60 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f366c00bca0 con 0x7f36840cff10 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f366c004d40 0x7f366c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3683fff640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f366c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f36837fe640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f366c003f60 0x7f366c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 -- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 msgr2=0x7f366c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36840cff10 0x7f366c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 -- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f366c003f60 msgr2=0x7f366c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f366c003f60 0x7f366c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.336+0000 7f3688c62640 1 -- 172.21.15.70:0/2585471735 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f366c003520 con 0x7f366c004d40 2024-04-15T15:16:03.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f3688c62640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f366c004d40 0x7f366c0051c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f367c009fc0 tx=0x7f366c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2a86b261386d38e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c039070 con 0x7f366c004d40 2024-04-15T15:16:03.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f367c02cc20 con 0x7f366c004d40 2024-04-15T15:16:03.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f3682ffd640 1 -- 172.21.15.70:0/2585471735 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c009c10 con 0x7f366c004d40 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f368a6c5640 1 -- 172.21.15.70:0/2585471735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f366c004d40 msgr2=0x7f366c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2585471735 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f366c004d40 0x7f366c0051c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f367c009fc0 tx=0x7f366c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f3688c62640 1 -- 172.21.15.70:0/2585471735 reap_dead start 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f368a6c5640 1 -- 172.21.15.70:0/2585471735 shutdown_connections 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.337+0000 7f368a6c5640 1 -- 172.21.15.70:0/2585471735 >> 172.21.15.70:0/2585471735 conn(0x7f36840b6870 msgr2=0x7f36840b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.338+0000 7f368a6c5640 1 -- 172.21.15.70:0/2585471735 shutdown_connections 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.338+0000 7f368a6c5640 1 -- 172.21.15.70:0/2585471735 wait complete. 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.339+0000 7f368a6c5640 1 Processor -- start 2024-04-15T15:16:03.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.339+0000 7f368a6c5640 1 -- start start 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f368414f710 0x7f368406f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 0x7f36841f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f36840cef00 con 0x7f3684141f10 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3684160180 con 0x7f368414f710 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f36840b0790 con 0x7f36841f5cf0 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41556/0 (socket says 172.21.15.70:41556) 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 -- 172.21.15.70:0/2327895050 learned_addr learned my addr 172.21.15.70:0/2327895050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3688c62640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 0x7f36841f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f36837fe640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f368414f710 0x7f368406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 -- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 msgr2=0x7f36841f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 0x7f36841f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 -- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f368414f710 msgr2=0x7f368406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f368414f710 0x7f368406f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 -- 172.21.15.70:0/2327895050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36841f66e0 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3688c62640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 0x7f36841f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3683fff640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3674007290 tx=0x7f3674007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367400f920 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3674018070 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367400fd40 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36841f69b0 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.340+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f368414ef10 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.341+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36840b1f40 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.342+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f367401c410 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.342+0000 7f3680ff9640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 0x7f367006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.342+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f367408bc80 con 0x7f3684141f10 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.342+0000 7f36837fe640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 0x7f367006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.343+0000 7f36837fe640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 0x7f367006ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3684070260 tx=0x7f3678009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.347+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3674059cb0 con 0x7f3684141f10 2024-04-15T15:16:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.521+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f36840d1810 con 0x7f3684141f10 2024-04-15T15:16:03.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.521+0000 7f3680ff9640 1 -- 172.21.15.70:0/2327895050 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f36740595b0 con 0x7f3684141f10 2024-04-15T15:16:03.523 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 msgr2=0x7f367006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.525 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 0x7f367006ede0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3684070260 tx=0x7f3678009040 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 msgr2=0x7f368406f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3674007290 tx=0x7f3674007750 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 shutdown_connections 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f36841f5cf0 0x7f36841f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f368414f710 0x7f368406f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f367006c920 0x7f367006ede0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 --2- 172.21.15.70:0/2327895050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3684141f10 0x7f368406f440 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 >> 172.21.15.70:0/2327895050 conn(0x7f36840b6870 msgr2=0x7f36840b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 shutdown_connections 2024-04-15T15:16:03.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.523+0000 7f368a6c5640 1 -- 172.21.15.70:0/2327895050 wait complete. 2024-04-15T15:16:03.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:03.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.640+0000 7f67fca0d640 1 Processor -- start 2024-04-15T15:16:03.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.640+0000 7f67fca0d640 1 -- start start 2024-04-15T15:16:03.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67fca0d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67f815e790 0x7f67f80b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67fca0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f80d1cf0 0x7f67f80d4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67fca0d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67f815eb70 con 0x7f67f80b7be0 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67fca0d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f67f80d3790 con 0x7f67f815e790 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67fca0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f67f80d45b0 con 0x7f67f80d1cf0 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67f659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67f815e790 0x7f67f80b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67f659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f67f815e790 0x7f67f80b76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41560/0 (socket says 172.21.15.70:41560) 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.641+0000 7f67f659c640 1 -- 172.21.15.70:0/3172899840 learned_addr learned my addr 172.21.15.70:0/3172899840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f6d9d640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f80d1cf0 0x7f67f80d4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f659c640 1 -- 172.21.15.70:0/3172899840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f80d1cf0 msgr2=0x7f67f80d4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f659c640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f80d1cf0 0x7f67f80d4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f659c640 1 -- 172.21.15.70:0/3172899840 >> v1:172.21.15.96:6789/0 conn(0x7f67f80b7be0 legacy=0x7f67f80d15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f659c640 1 -- 172.21.15.70:0/3172899840 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f80d3c80 con 0x7f67f815e790 2024-04-15T15:16:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.642+0000 7f67f659c640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f815e790 0x7f67f80b76a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f67f80d3910 tx=0x7f67e802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8627472b60c7e894 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67f559a640 1 -- 172.21.15.70:0/3172899840 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e802c9c0 con 0x7f67f815e790 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67f559a640 1 -- 172.21.15.70:0/3172899840 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67e802cb60 con 0x7f67f815e790 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67f559a640 1 -- 172.21.15.70:0/3172899840 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e80343f0 con 0x7f67f815e790 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 -- 172.21.15.70:0/3172899840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f815e790 msgr2=0x7f67f80b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f815e790 0x7f67f80b76a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f67f80d3910 tx=0x7f67e802be20 comp rx=0 tx=0).stop 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 -- 172.21.15.70:0/3172899840 shutdown_connections 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f80d1cf0 0x7f67f80d4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 --2- 172.21.15.70:0/3172899840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f815e790 0x7f67f80b76a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 -- 172.21.15.70:0/3172899840 >> 172.21.15.70:0/3172899840 conn(0x7f67f8067b40 msgr2=0x7f67f8067f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 -- 172.21.15.70:0/3172899840 shutdown_connections 2024-04-15T15:16:03.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.643+0000 7f67fca0d640 1 -- 172.21.15.70:0/3172899840 wait complete. 2024-04-15T15:16:03.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 Processor -- start 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 -- start start 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67f80b7be0 0x7f67f81f8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f815e790 0x7f67f80cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80d4f80 con 0x7f67f80d1cf0 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f67f80d4240 con 0x7f67f80b7be0 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67fca0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f67f80d5100 con 0x7f67f815e790 2024-04-15T15:16:03.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.645+0000 7f67f5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41572/0 (socket says 172.21.15.70:41572) 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 -- 172.21.15.70:0/681385534 learned_addr learned my addr 172.21.15.70:0/681385534 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f6d9d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f815e790 0x7f67f80cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f659c640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67f80b7be0 0x7f67f81f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 -- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f815e790 msgr2=0x7f67f80cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f815e790 0x7f67f80cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 -- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67f80b7be0 msgr2=0x7f67f81f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67f80b7be0 0x7f67f81f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 -- 172.21.15.70:0/681385534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f8160610 con 0x7f67f80d1cf0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67f5d9b640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c400 tx=0x7f67ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec017070 con 0x7f67f80d1cf0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67ec0052d0 con 0x7f67f80d1cf0 2024-04-15T15:16:03.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f81608f0 con 0x7f67f80d1cf0 2024-04-15T15:16:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec003850 con 0x7f67f80d1cf0 2024-04-15T15:16:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.646+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f8161190 con 0x7f67f80d1cf0 2024-04-15T15:16:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.647+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f8161390 con 0x7f67f80d1cf0 2024-04-15T15:16:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.648+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f67ec01a070 con 0x7f67f80d1cf0 2024-04-15T15:16:03.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.648+0000 7f67e6ffd640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.648+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f67ec08b240 con 0x7f67f80d1cf0 2024-04-15T15:16:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.648+0000 7f67f659c640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.648+0000 7f67f659c640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 0x7f67d806ede0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f67e8035000 tx=0x7f67e8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.652+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f67f8161390 con 0x7f67f80d1cf0 2024-04-15T15:16:03.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.825+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f67f80689e0 con 0x7f67f80d1cf0 2024-04-15T15:16:03.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.825+0000 7f67e6ffd640 1 -- 172.21.15.70:0/681385534 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f67ec058c50 con 0x7f67f80d1cf0 2024-04-15T15:16:03.827 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 msgr2=0x7f67d806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 0x7f67d806ede0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f67e8035000 tx=0x7f67e8030000 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 msgr2=0x7f67f815fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c400 tx=0x7f67ec00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 shutdown_connections 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f67f815e790 0x7f67f80cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f67d806c920 0x7f67d806ede0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f67f80d1cf0 0x7f67f815fdc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 --2- 172.21.15.70:0/681385534 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f67f80b7be0 0x7f67f81f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.830 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 >> 172.21.15.70:0/681385534 conn(0x7f67f8067b40 msgr2=0x7f67f80b5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 shutdown_connections 2024-04-15T15:16:03.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.827+0000 7f67fca0d640 1 -- 172.21.15.70:0/681385534 wait complete. 2024-04-15T15:16:03.840 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.945+0000 7f0c168c5640 1 Processor -- start 2024-04-15T15:16:03.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.945+0000 7f0c168c5640 1 -- start start 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c168c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 0x7f0c100b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c168c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c100b4350 0x7f0c100b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c168c5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c100b10d0 con 0x7f0c100b1b60 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c168c5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0c1015f090 con 0x7f0c100b4350 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c168c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0c100cdf80 con 0x7f0c1015ecb0 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c0f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c100b4350 0x7f0c100b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c0f7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0c100b4350 0x7f0c100b1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41588/0 (socket says 172.21.15.70:41588) 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.946+0000 7f0c0f7fe640 1 -- 172.21.15.70:0/76374499 learned_addr learned my addr 172.21.15.70:0/76374499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0ffff640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 0x7f0c100b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0f7fe640 1 -- 172.21.15.70:0/76374499 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 msgr2=0x7f0c100b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0f7fe640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 0x7f0c100b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0f7fe640 1 -- 172.21.15.70:0/76374499 >> v1:172.21.15.96:6789/0 conn(0x7f0c100b1b60 legacy=0x7f0c100ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0f7fe640 1 -- 172.21.15.70:0/76374499 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c100af920 con 0x7f0c100b4350 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0ffff640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 0x7f0c100b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.947+0000 7f0c0f7fe640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c100b4350 0x7f0c100b1620 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009870 tx=0x7f0c0402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2cae3782112118 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c0effd640 1 -- 172.21.15.70:0/76374499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04034070 con 0x7f0c100b4350 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c0effd640 1 -- 172.21.15.70:0/76374499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c04030840 con 0x7f0c100b4350 2024-04-15T15:16:03.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c0effd640 1 -- 172.21.15.70:0/76374499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c04030c10 con 0x7f0c100b4350 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 -- 172.21.15.70:0/76374499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c100b4350 msgr2=0x7f0c100b1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c100b4350 0x7f0c100b1620 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009870 tx=0x7f0c0402b980 comp rx=0 tx=0).stop 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 -- 172.21.15.70:0/76374499 shutdown_connections 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c100b4350 0x7f0c100b1620 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 --2- 172.21.15.70:0/76374499 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c1015ecb0 0x7f0c100b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 -- 172.21.15.70:0/76374499 >> 172.21.15.70:0/76374499 conn(0x7f0c100b6870 msgr2=0x7f0c100b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 -- 172.21.15.70:0/76374499 shutdown_connections 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.948+0000 7f0c168c5640 1 -- 172.21.15.70:0/76374499 wait complete. 2024-04-15T15:16:03.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.949+0000 7f0c168c5640 1 Processor -- start 2024-04-15T15:16:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 -- start start 2024-04-15T15:16:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c100b1b60 0x7f0c1015c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c100b4350 0x7f0c1015f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0c100b4790 con 0x7f0c1015ecb0 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0c100afee0 con 0x7f0c100b1b60 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c168c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0c100b4910 con 0x7f0c100b4350 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c14e62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c14e62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41590/0 (socket says 172.21.15.70:41590) 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.950+0000 7f0c14e62640 1 -- 172.21.15.70:0/2413624183 learned_addr learned my addr 172.21.15.70:0/2413624183 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c0f7fe640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c100b4350 0x7f0c1015f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c0ffff640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c100b1b60 0x7f0c1015c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 -- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c100b4350 msgr2=0x7f0c1015f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c100b4350 0x7f0c1015f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 -- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c100b1b60 msgr2=0x7f0c1015c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c100b1b60 0x7f0c1015c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 -- 172.21.15.70:0/2413624183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c10160140 con 0x7f0c1015ecb0 2024-04-15T15:16:03.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c14e62640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c7b0 tx=0x7f0c0800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800ce60 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c08004b10 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800a850 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1015fb80 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.951+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c10160490 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.952+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c100689e0 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.953+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0c0800a9f0 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.953+0000 7f0c0cff9640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 0x7f0bfc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.953+0000 7f0c0ffff640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 0x7f0bfc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.953+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0c0808bf80 con 0x7f0c1015ecb0 2024-04-15T15:16:03.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.954+0000 7f0c0ffff640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 0x7f0bfc06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c100cef50 tx=0x7f0c00008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:03.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:03.958+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0c08020070 con 0x7f0c1015ecb0 2024-04-15T15:16:04.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.135+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0c100d2170 con 0x7f0c1015ecb0 2024-04-15T15:16:04.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.136+0000 7f0c0cff9640 1 -- 172.21.15.70:0/2413624183 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f0c080599f0 con 0x7f0c1015ecb0 2024-04-15T15:16:04.138 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 msgr2=0x7f0bfc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 0x7f0bfc06edb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c100cef50 tx=0x7f0c00008040 comp rx=0 tx=0).stop 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 msgr2=0x7f0c1014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0c0800c7b0 tx=0x7f0c0800cc70 comp rx=0 tx=0).stop 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 shutdown_connections 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0bfc06c8f0 0x7f0bfc06edb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0c1015ecb0 0x7f0c1014d980 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0c100b4350 0x7f0c1015f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 --2- 172.21.15.70:0/2413624183 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0c100b1b60 0x7f0c1015c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 >> 172.21.15.70:0/2413624183 conn(0x7f0c100b6870 msgr2=0x7f0c10074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 shutdown_connections 2024-04-15T15:16:04.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.138+0000 7f0c168c5640 1 -- 172.21.15.70:0/2413624183 wait complete. 2024-04-15T15:16:04.151 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:04.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.255+0000 7f0a5ab82640 1 Processor -- start 2024-04-15T15:16:04.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.255+0000 7f0a5ab82640 1 -- start start 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5ab82640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a5415e700 0x7f0a540b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5ab82640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 0x7f0a540cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5ab82640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a540b47a0 con 0x7f0a540d2360 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5ab82640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0a540b53b0 con 0x7f0a5415e700 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5ab82640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540b4a80 con 0x7f0a540d3270 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5891e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a5415e700 0x7f0a540b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5891e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0a5415e700 0x7f0a540b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41592/0 (socket says 172.21.15.70:41592) 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.256+0000 7f0a5891e640 1 -- 172.21.15.70:0/2544579123 learned_addr learned my addr 172.21.15.70:0/2544579123 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5911f640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 0x7f0a540cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5891e640 1 -- 172.21.15.70:0/2544579123 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 msgr2=0x7f0a540cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5891e640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 0x7f0a540cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5891e640 1 -- 172.21.15.70:0/2544579123 >> v1:172.21.15.96:6789/0 conn(0x7f0a540d2360 legacy=0x7f0a540d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5891e640 1 -- 172.21.15.70:0/2544579123 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540b4d60 con 0x7f0a5415e700 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5911f640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 0x7f0a540cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.257+0000 7f0a5891e640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a540b5840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009850 tx=0x7f0a4802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7595e3353099ced3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a537fe640 1 -- 172.21.15.70:0/2544579123 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48004230 con 0x7f0a5415e700 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a537fe640 1 -- 172.21.15.70:0/2544579123 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a480043d0 con 0x7f0a5415e700 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a537fe640 1 -- 172.21.15.70:0/2544579123 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48003810 con 0x7f0a5415e700 2024-04-15T15:16:04.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 -- 172.21.15.70:0/2544579123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 msgr2=0x7f0a540b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a540b5840 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0a48009850 tx=0x7f0a4802b960 comp rx=0 tx=0).stop 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 -- 172.21.15.70:0/2544579123 shutdown_connections 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d3270 0x7f0a540cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/2544579123 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a540b5840 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 -- 172.21.15.70:0/2544579123 >> 172.21.15.70:0/2544579123 conn(0x7f0a5408bf60 msgr2=0x7f0a540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 -- 172.21.15.70:0/2544579123 shutdown_connections 2024-04-15T15:16:04.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.258+0000 7f0a5ab82640 1 -- 172.21.15.70:0/2544579123 wait complete. 2024-04-15T15:16:04.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.259+0000 7f0a5ab82640 1 Processor -- start 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.259+0000 7f0a5ab82640 1 -- start start 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 0x7f0a540cfa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d3270 0x7f0a541603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b5d80 con 0x7f0a5415e700 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0a540b4450 con 0x7f0a540d3270 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5ab82640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0a540d1d40 con 0x7f0a540d2360 2024-04-15T15:16:04.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5911f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5911f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41594/0 (socket says 172.21.15.70:41594) 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5911f640 1 -- 172.21.15.70:0/3235604112 learned_addr learned my addr 172.21.15.70:0/3235604112 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.260+0000 7f0a5891e640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 0x7f0a540cfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a53fff640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d3270 0x7f0a541603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 -- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 msgr2=0x7f0a540cfa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 0x7f0a540cfa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 -- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d3270 msgr2=0x7f0a541603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d3270 0x7f0a541603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 -- 172.21.15.70:0/3235604112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540b1360 con 0x7f0a5415e700 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5891e640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 0x7f0a540cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5911f640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c007ef0 tx=0x7f0a4c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c01a070 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c003b90 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5415eae0 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c004030 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.261+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a5415f380 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.262+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a540689e0 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.262+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0a4c01d070 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.263+0000 7f0a517fa640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.263+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0a4c08c0a0 con 0x7f0a5415e700 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.263+0000 7f0a5891e640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.263+0000 7f0a5891e640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 0x7f0a3806edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a48035000 tx=0x7f0a48008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.267+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0a4c010070 con 0x7f0a5415e700 2024-04-15T15:16:04.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.442+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0a540ba500 con 0x7f0a5415e700 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.443+0000 7f0a517fa640 1 -- 172.21.15.70:0/3235604112 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f0a4c059b10 con 0x7f0a5415e700 2024-04-15T15:16:04.445 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 msgr2=0x7f0a3806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 0x7f0a3806edb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a48035000 tx=0x7f0a48008030 comp rx=0 tx=0).stop 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 msgr2=0x7f0a54159420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c007ef0 tx=0x7f0a4c00d4a0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 shutdown_connections 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0a3806c8f0 0x7f0a3806edb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0a5415e700 0x7f0a54159420 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0a540d3270 0x7f0a541603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 --2- 172.21.15.70:0/3235604112 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0a540d2360 0x7f0a540cfa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 >> 172.21.15.70:0/3235604112 conn(0x7f0a5408bf60 msgr2=0x7f0a540c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 shutdown_connections 2024-04-15T15:16:04.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.445+0000 7f0a5ab82640 1 -- 172.21.15.70:0/3235604112 wait complete. 2024-04-15T15:16:04.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:04.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.560+0000 7f4a80500640 1 Processor -- start 2024-04-15T15:16:04.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.561+0000 7f4a80500640 1 -- start start 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.561+0000 7f4a80500640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a780d0690 0x7f4a780d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.561+0000 7f4a80500640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a780d0fb0 con 0x7f4a780b3160 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a80500640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a7815cb50 con 0x7f4a780ae280 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a80500640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4a7815bda0 con 0x7f4a780d0690 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7e29c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a780d0690 0x7f4a780d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7da9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4a780b3160 0x7f4a7815b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51328/0 (socket says 172.21.15.70:51328) 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7e29c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a780d0690 0x7f4a780d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41598/0 (socket says 172.21.15.70:41598) 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7da9b640 1 -- 172.21.15.70:0/1061437363 learned_addr learned my addr 172.21.15.70:0/1061437363 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1061437363 >> v1:172.21.15.96:6789/0 conn(0x7f4a780ae280 legacy=0x7f4a780ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1061437363 >> v1:172.21.15.70:6789/0 conn(0x7f4a780b3160 legacy=0x7f4a7815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.562+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1061437363 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7815bfd0 con 0x7f4a780d0690 2024-04-15T15:16:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a7e29c640 1 --2- 172.21.15.70:0/1061437363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780d0690 0x7f4a780d0a70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4a700097a0 tx=0x7f4a7002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7fbec61d8cbe85d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a7d29a640 1 -- 172.21.15.70:0/1061437363 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a70035020 con 0x7f4a780d0690 2024-04-15T15:16:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a7d29a640 1 -- 172.21.15.70:0/1061437363 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a70031830 con 0x7f4a780d0690 2024-04-15T15:16:04.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a7d29a640 1 -- 172.21.15.70:0/1061437363 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a70031c00 con 0x7f4a780d0690 2024-04-15T15:16:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a80500640 1 -- 172.21.15.70:0/1061437363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780d0690 msgr2=0x7f4a780d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a80500640 1 --2- 172.21.15.70:0/1061437363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780d0690 0x7f4a780d0a70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4a700097a0 tx=0x7f4a7002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a80500640 1 -- 172.21.15.70:0/1061437363 shutdown_connections 2024-04-15T15:16:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a80500640 1 --2- 172.21.15.70:0/1061437363 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780d0690 0x7f4a780d0a70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.563+0000 7f4a80500640 1 -- 172.21.15.70:0/1061437363 >> 172.21.15.70:0/1061437363 conn(0x7f4a7806be00 msgr2=0x7f4a7806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.564+0000 7f4a80500640 1 -- 172.21.15.70:0/1061437363 shutdown_connections 2024-04-15T15:16:04.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.564+0000 7f4a80500640 1 -- 172.21.15.70:0/1061437363 wait complete. 2024-04-15T15:16:04.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 Processor -- start 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 -- start start 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a780b3160 0x7f4a78099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a780d0690 0x7f4a78099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7815c870 con 0x7f4a780ae280 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7815c590 con 0x7f4a780b3160 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a80500640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4a780b4fb0 con 0x7f4a780d0690 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a7e29c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a7e29c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41606/0 (socket says 172.21.15.70:41606) 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.565+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1941016244 learned_addr learned my addr 172.21.15.70:0/1941016244 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a780d0690 msgr2=0x7f4a78099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a780d0690 0x7f4a78099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a780b3160 msgr2=0x7f4a78099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a780b3160 0x7f4a78099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 -- 172.21.15.70:0/1941016244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a781e4d90 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a7e29c640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4a70000c00 tx=0x7f4a700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a70040070 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a70039a00 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a781e5070 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a70035040 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.566+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a781e5630 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.567+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a780689e0 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.567+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4a7004b060 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.568+0000 7f4a76ffd640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 0x7f4a6406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.568+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4a700ad970 con 0x7f4a780ae280 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.568+0000 7f4a7da9b640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 0x7f4a6406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.568+0000 7f4a7da9b640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 0x7f4a6406edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a781572c0 tx=0x7f4a6c007a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.572+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a7007b9a0 con 0x7f4a780ae280 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.758+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4a780739e0 con 0x7f4a780ae280 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.758+0000 7f4a76ffd640 1 -- 172.21.15.70:0/1941016244 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f4a7007b2a0 con 0x7f4a780ae280 2024-04-15T15:16:04.760 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 msgr2=0x7f4a6406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 0x7f4a6406edb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a781572c0 tx=0x7f4a6c007a40 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 msgr2=0x7f4a7815a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f4a70000c00 tx=0x7f4a700046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 shutdown_connections 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a780d0690 0x7f4a78099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a780b3160 0x7f4a78099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a6406c8f0 0x7f4a6406edb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 --2- 172.21.15.70:0/1941016244 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a780ae280 0x7f4a7815a3f0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 >> 172.21.15.70:0/1941016244 conn(0x7f4a7806be00 msgr2=0x7f4a780b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 shutdown_connections 2024-04-15T15:16:04.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.760+0000 7f4a80500640 1 -- 172.21.15.70:0/1941016244 wait complete. 2024-04-15T15:16:04.773 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:04.773 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:04.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:04.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.929+0000 7fae01414640 1 Processor -- start 2024-04-15T15:16:04.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.929+0000 7fae01414640 1 -- start start 2024-04-15T15:16:04.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fae01414640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 0x7fadfc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fae01414640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fadfc0d2360 0x7fadfc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fae01414640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fadfc0b47a0 con 0x7fadfc15eae0 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fae01414640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fadfc0b53b0 con 0x7fadfc0d2360 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fae01414640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fadfc0b4a80 con 0x7fadfc15e700 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfa7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fadfc0d2360 0x7fadfc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fadfc15eae0 0x7fadfc0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51338/0 (socket says 172.21.15.70:51338) 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfb7fe640 1 -- 172.21.15.70:0/3806122992 learned_addr learned my addr 172.21.15.70:0/3806122992 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfaffd640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 0x7fadfc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfa7fc640 1 -- 172.21.15.70:0/3806122992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 msgr2=0x7fadfc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfa7fc640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 0x7fadfc0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfa7fc640 1 -- 172.21.15.70:0/3806122992 >> v1:172.21.15.70:6789/0 conn(0x7fadfc15eae0 legacy=0x7fadfc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfa7fc640 1 -- 172.21.15.70:0/3806122992 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc0b4d60 con 0x7fadfc0d2360 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadf9ffb640 1 -- 172.21.15.70:0/3806122992 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4210574335 0 0) 0x7fadfc0b47a0 con 0x7fadfc15eae0 2024-04-15T15:16:04.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.930+0000 7fadfaffd640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 0x7fadfc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:04.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.931+0000 7fadfa7fc640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc0d2360 0x7fadfc0d2b60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fadf00097a0 tx=0x7fadf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e37153ccbdc9e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.931+0000 7fadf9ffb640 1 -- 172.21.15.70:0/3806122992 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0035020 con 0x7fadfc0d2360 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.931+0000 7fadf9ffb640 1 -- 172.21.15.70:0/3806122992 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf0031830 con 0x7fadfc0d2360 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.931+0000 7fadf9ffb640 1 -- 172.21.15.70:0/3806122992 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0031c00 con 0x7fadfc0d2360 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 -- 172.21.15.70:0/3806122992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc0d2360 msgr2=0x7fadfc0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc0d2360 0x7fadfc0d2b60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fadf00097a0 tx=0x7fadf002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 -- 172.21.15.70:0/3806122992 shutdown_connections 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc0d2360 0x7fadfc0d2b60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 --2- 172.21.15.70:0/3806122992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc15e700 0x7fadfc0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 -- 172.21.15.70:0/3806122992 >> 172.21.15.70:0/3806122992 conn(0x7fadfc08bf60 msgr2=0x7fadfc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:04.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 -- 172.21.15.70:0/3806122992 shutdown_connections 2024-04-15T15:16:04.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.932+0000 7fae01414640 1 -- 172.21.15.70:0/3806122992 wait complete. 2024-04-15T15:16:04.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.933+0000 7fae01414640 1 Processor -- start 2024-04-15T15:16:04.936 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.933+0000 7fae01414640 1 -- start start 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc0d2360 0x7fadfc152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadfc15eae0 0x7fadfc15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc0b5d80 con 0x7fadfc15e700 2024-04-15T15:16:04.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fadfc0b4450 con 0x7fadfc15eae0 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fae01414640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fadfc0d1d40 con 0x7fadfc0d2360 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fadfa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fadfa7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41618/0 (socket says 172.21.15.70:41618) 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fadfa7fc640 1 -- 172.21.15.70:0/1845195813 learned_addr learned my addr 172.21.15.70:0/1845195813 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fadfaffd640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc0d2360 0x7fadfc152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.934+0000 7fadfb7fe640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadfc15eae0 0x7fadfc15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 -- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc0d2360 msgr2=0x7fadfc152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc0d2360 0x7fadfc152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 -- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadfc15eae0 msgr2=0x7fadfc15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadfc15eae0 0x7fadfc15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 -- 172.21.15.70:0/1845195813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadfc1f7d60 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fadfa7fc640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009890 tx=0x7fadf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0041070 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf003b920 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadfc1f8040 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0035040 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.935+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadfc1f8600 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.936+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadfc0689e0 con 0x7fadfc15e700 2024-04-15T15:16:04.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.936+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fadf004c060 con 0x7fadfc15e700 2024-04-15T15:16:04.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.937+0000 7fade37fe640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 0x7fade806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:04.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.937+0000 7fadfaffd640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 0x7fade806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:04.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.937+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fadf00ad760 con 0x7fadfc15e700 2024-04-15T15:16:04.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.938+0000 7fadfaffd640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 0x7fade806edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fadec003b80 tx=0x7fadec0098e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:04.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:04.941+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fadf007b150 con 0x7fadfc15e700 2024-04-15T15:16:05.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.117+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fadfc0ba240 con 0x7fadfc15e700 2024-04-15T15:16:05.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.118+0000 7fade37fe640 1 -- 172.21.15.70:0/1845195813 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fadf007f450 con 0x7fadfc15e700 2024-04-15T15:16:05.121 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:05.121 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:05.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 msgr2=0x7fade806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 0x7fade806edb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fadec003b80 tx=0x7fadec0098e0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 msgr2=0x7fadfc0ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fadf0009890 tx=0x7fadf00046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 shutdown_connections 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fadfc15eae0 0x7fadfc15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fade806c8f0 0x7fade806edb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fadfc15e700 0x7fadfc0ccac0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.120+0000 7fae01414640 1 --2- 172.21.15.70:0/1845195813 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fadfc0d2360 0x7fadfc152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.121+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 >> 172.21.15.70:0/1845195813 conn(0x7fadfc08bf60 msgr2=0x7fadfc06e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.121+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 shutdown_connections 2024-04-15T15:16:05.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.121+0000 7fae01414640 1 -- 172.21.15.70:0/1845195813 wait complete. 2024-04-15T15:16:05.134 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:05.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:05.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.238+0000 7fbfd21db640 1 Processor -- start 2024-04-15T15:16:05.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.238+0000 7fbfd21db640 1 -- start start 2024-04-15T15:16:05.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfd21db640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fbfcc0d0690 0x7fbfcc0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfd21db640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfcc0ae280 0x7fbfcc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfd21db640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfcc0d0fb0 con 0x7fbfcc0b3160 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfd21db640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbfcc15bda0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfd21db640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fbfcc15cb50 con 0x7fbfcc0d0690 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfcaffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbfcc0b3160 0x7fbfcc15b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51354/0 (socket says 172.21.15.70:51354) 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfcaffd640 1 -- 172.21.15.70:0/2145986071 learned_addr learned my addr 172.21.15.70:0/2145986071 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfcbfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfcc0ae280 0x7fbfcc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfcb7fe640 1 --2- 172.21.15.70:0/2145986071 >> v2:172.21.15.96:3300/0 conn(0x7fbfcc0d0690 0x7fbfcc0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.239+0000 7fbfcbfff640 1 -- 172.21.15.70:0/2145986071 >> v2:172.21.15.96:3300/0 conn(0x7fbfcc0d0690 msgr2=0x7fbfcc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfcbfff640 1 --2- 172.21.15.70:0/2145986071 >> v2:172.21.15.96:3300/0 conn(0x7fbfcc0d0690 0x7fbfcc0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfcbfff640 1 -- 172.21.15.70:0/2145986071 >> v1:172.21.15.70:6789/0 conn(0x7fbfcc0b3160 legacy=0x7fbfcc15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfcbfff640 1 -- 172.21.15.70:0/2145986071 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfcc15bfd0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfcbfff640 1 --2- 172.21.15.70:0/2145986071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc0ae710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc06f160 tx=0x7fbfc402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1228e2fdabf9aa0e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfca7fc640 1 -- 172.21.15.70:0/2145986071 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4039070 con 0x7fbfcc0ae280 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfca7fc640 1 -- 172.21.15.70:0/2145986071 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfc402cc80 con 0x7fbfcc0ae280 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.240+0000 7fbfca7fc640 1 -- 172.21.15.70:0/2145986071 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc4034660 con 0x7fbfcc0ae280 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 -- 172.21.15.70:0/2145986071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 msgr2=0x7fbfcc0ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 --2- 172.21.15.70:0/2145986071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc0ae710 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc06f160 tx=0x7fbfc402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 -- 172.21.15.70:0/2145986071 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 --2- 172.21.15.70:0/2145986071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc0ae710 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 --2- 172.21.15.70:0/2145986071 >> v2:172.21.15.96:3300/0 conn(0x7fbfcc0d0690 0x7fbfcc0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 -- 172.21.15.70:0/2145986071 >> 172.21.15.70:0/2145986071 conn(0x7fbfcc06be00 msgr2=0x7fbfcc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 -- 172.21.15.70:0/2145986071 shutdown_connections 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.241+0000 7fbfd21db640 1 -- 172.21.15.70:0/2145986071 wait complete. 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.242+0000 7fbfd21db640 1 Processor -- start 2024-04-15T15:16:05.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.242+0000 7fbfd21db640 1 -- start start 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 0x7fbfcc09a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfcc0d0690 0x7fbfcc1566f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc15c870 con 0x7fbfcc0ae280 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbfcc15c590 con 0x7fbfcc0d0690 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfd21db640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfcc0b4fb0 con 0x7fbfcc0b3160 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41630/0 (socket says 172.21.15.70:41630) 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcb7fe640 1 -- 172.21.15.70:0/627816657 learned_addr learned my addr 172.21.15.70:0/627816657 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcaffd640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 0x7fbfcc09a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcbfff640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfcc0d0690 0x7fbfcc1566f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcb7fe640 1 -- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 msgr2=0x7fbfcc09a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.243+0000 7fbfcb7fe640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 0x7fbfcc09a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfcb7fe640 1 -- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfcc0d0690 msgr2=0x7fbfcc1566f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfcb7fe640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfcc0d0690 0x7fbfcc1566f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfcb7fe640 1 -- 172.21.15.70:0/627816657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfcc09c230 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfcaffd640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 0x7fbfcc09a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfcb7fe640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00c7f0 tx=0x7fbfbc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc015070 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfbc004240 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc007620 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfcc09c510 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.244+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfcc09cad0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.245+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfcc0689e0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.245+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbfbc0079d0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.246+0000 7fbfb3fff640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 0x7fbfb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.246+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbfbc08c350 con 0x7fbfcc0ae280 2024-04-15T15:16:05.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.246+0000 7fbfcaffd640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 0x7fbfb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.246+0000 7fbfcaffd640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 0x7fbfb806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc1587f0 tx=0x7fbfc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.250+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbfbc05a410 con 0x7fbfcc0ae280 2024-04-15T15:16:05.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.426+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbfcc0739e0 con 0x7fbfcc0ae280 2024-04-15T15:16:05.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.427+0000 7fbfb3fff640 1 -- 172.21.15.70:0/627816657 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbfbc059d10 con 0x7fbfcc0ae280 2024-04-15T15:16:05.429 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 msgr2=0x7fbfb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 0x7fbfb806edb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc1587f0 tx=0x7fbfc0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 msgr2=0x7fbfcc148600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc00c7f0 tx=0x7fbfbc00ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 shutdown_connections 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfcc0d0690 0x7fbfcc1566f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfcc0b3160 0x7fbfcc09a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbfb806c8f0 0x7fbfb806edb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 --2- 172.21.15.70:0/627816657 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfcc0ae280 0x7fbfcc148600 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 >> 172.21.15.70:0/627816657 conn(0x7fbfcc06be00 msgr2=0x7fbfcc0b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 shutdown_connections 2024-04-15T15:16:05.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.429+0000 7fbfd21db640 1 -- 172.21.15.70:0/627816657 wait complete. 2024-04-15T15:16:05.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:05.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.546+0000 7f55b0cc8640 1 Processor -- start 2024-04-15T15:16:05.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.546+0000 7f55b0cc8640 1 -- start start 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55b0cc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55ac0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55b0cc8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55ac0d2360 0x7f55ac0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55b0cc8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55ac0b47a0 con 0x7f55ac15eae0 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55b0cc8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f55ac0b53b0 con 0x7f55ac0d2360 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55b0cc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f55ac0b4a80 con 0x7f55ac15e700 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55aad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f55ac15eae0 0x7f55ac0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51360/0 (socket says 172.21.15.70:51360) 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55aad9d640 1 -- 172.21.15.70:0/2124768655 learned_addr learned my addr 172.21.15.70:0/2124768655 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> v2:172.21.15.96:3300/0 conn(0x7f55ac0d2360 0x7f55ac0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55aa59c640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55ac0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1836028950 0 0) 0x7f55ac0b47a0 con 0x7f55ac15eae0 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5594003520 con 0x7f55ac15eae0 2024-04-15T15:16:05.552 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3762606534 0 0) 0x7f5594003520 con 0x7f55ac15eae0 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 msgr2=0x7f55ac0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55ac0b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 >> v2:172.21.15.96:3300/0 conn(0x7f55ac0d2360 msgr2=0x7f55ac0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 --2- 172.21.15.70:0/2124768655 >> v2:172.21.15.96:3300/0 conn(0x7f55ac0d2360 0x7f55ac0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ac0b4d60 con 0x7f55ac15eae0 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f55a40019f0 con 0x7f55ac15eae0 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> v2:172.21.15.96:3300/0 conn(0x7f55ac0d2360 0x7f55ac0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 >> v1:172.21.15.70:6789/0 conn(0x7f55ac15eae0 legacy=0x7f55ac0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55940039b0 0x7f5594003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5594004800 0x7f5594004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0b53b0 con 0x7f5594004800 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0b4a80 con 0x7f55940039b0 2024-04-15T15:16:05.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f559400b9c0 con 0x7f55ac15e700 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.547+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5594004800 0x7f5594004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55aa59c640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55940039b0 0x7f5594003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55aad9d640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 -- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 msgr2=0x7f55940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 -- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55940039b0 msgr2=0x7f5594003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55940039b0 0x7f5594003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 -- 172.21.15.70:0/2124768655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5594003520 con 0x7f5594004800 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55aad9d640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac15e700 0x7f55940051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.548+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5594004800 0x7f5594004c80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009c90 tx=0x7f5594005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c13bb6d569de6aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0039070 con 0x7f5594004800 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a002c980 con 0x7f5594004800 2024-04-15T15:16:05.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55a959a640 1 -- 172.21.15.70:0/2124768655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0009860 con 0x7f5594004800 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 -- 172.21.15.70:0/2124768655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5594004800 msgr2=0x7f5594004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/2124768655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5594004800 0x7f5594004c80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009c90 tx=0x7f5594005830 comp rx=0 tx=0).stop 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55aad9d640 1 -- 172.21.15.70:0/2124768655 reap_dead start 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 -- 172.21.15.70:0/2124768655 shutdown_connections 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 -- 172.21.15.70:0/2124768655 >> 172.21.15.70:0/2124768655 conn(0x7f55ac08bf60 msgr2=0x7f55ac0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 -- 172.21.15.70:0/2124768655 shutdown_connections 2024-04-15T15:16:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.549+0000 7f55b0cc8640 1 -- 172.21.15.70:0/2124768655 wait complete. 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 Processor -- start 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 -- start start 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 0x7f55ac0b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 0x7f55ac14c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0d1d40 con 0x7f55ac15ae20 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55ac0b4450 con 0x7f5594004800 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55b0cc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f55ac0d1ec0 con 0x7f55ac14bd60 2024-04-15T15:16:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55a9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55a9d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41650/0 (socket says 172.21.15.70:41650) 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.551+0000 7f55a9d9b640 1 -- 172.21.15.70:0/1899162460 learned_addr learned my addr 172.21.15.70:0/1899162460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 -- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 msgr2=0x7f55ac14c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55aad9d640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 0x7f55ac14c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55aa59c640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 0x7f55ac0b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 0x7f55ac14c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 -- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 msgr2=0x7f55ac0b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 0x7f55ac0b3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 -- 172.21.15.70:0/1899162460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55ac14ba80 con 0x7f55ac15ae20 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55aad9d640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 0x7f55ac14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55aa59c640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 0x7f55ac0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55a9d9b640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f55a002c530 tx=0x7f55a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0039050 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55a003a040 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0009860 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55ac0c81c0 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.552+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55ac0b2b50 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.553+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55ac0689e0 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.554+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f55a0047020 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.554+0000 7f559affd640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 0x7f558c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.554+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f55a00adb10 con 0x7f55ac15ae20 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.554+0000 7f55aa59c640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 0x7f558c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.555+0000 7f55aa59c640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 0x7f558c06edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f559c0099d0 tx=0x7f559c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.558+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f55ac14ba80 con 0x7f55ac15ae20 2024-04-15T15:16:05.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.733+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f55ac161de0 con 0x7f55ac15ae20 2024-04-15T15:16:05.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.733+0000 7f559affd640 1 -- 172.21.15.70:0/1899162460 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f55a007bc60 con 0x7f55ac15ae20 2024-04-15T15:16:05.735 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 msgr2=0x7f558c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 0x7f558c06edb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f559c0099d0 tx=0x7f559c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 msgr2=0x7f55ac14b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f55a002c530 tx=0x7f55a0004730 comp rx=0 tx=0).stop 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 shutdown_connections 2024-04-15T15:16:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55ac14bd60 0x7f55ac14c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f558c06c8f0 0x7f558c06edb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55ac15ae20 0x7f55ac14b540 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 --2- 172.21.15.70:0/1899162460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5594004800 0x7f55ac0b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 >> 172.21.15.70:0/1899162460 conn(0x7f55ac08bf60 msgr2=0x7f55ac0d00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 shutdown_connections 2024-04-15T15:16:05.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.735+0000 7f55b0cc8640 1 -- 172.21.15.70:0/1899162460 wait complete. 2024-04-15T15:16:05.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:05.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.852+0000 7fc2adaf2640 1 Processor -- start 2024-04-15T15:16:05.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.852+0000 7fc2adaf2640 1 -- start start 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2adaf2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2a815ecb0 0x7fc2a80b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2adaf2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc2a80b4350 0x7fc2a80b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2adaf2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2a80b10d0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2adaf2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a815f090 con 0x7fc2a815ecb0 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2adaf2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc2a80cdf80 con 0x7fc2a80b4350 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2a815ecb0 0x7fc2a80b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc2a815ecb0 0x7fc2a80b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41664/0 (socket says 172.21.15.70:41664) 2024-04-15T15:16:05.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 -- 172.21.15.70:0/3644334306 learned_addr learned my addr 172.21.15.70:0/3644334306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 -- 172.21.15.70:0/3644334306 >> v2:172.21.15.96:3300/0 conn(0x7fc2a80b4350 msgr2=0x7fc2a80b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 --2- 172.21.15.70:0/3644334306 >> v2:172.21.15.96:3300/0 conn(0x7fc2a80b4350 0x7fc2a80b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 -- 172.21.15.70:0/3644334306 >> v1:172.21.15.96:6789/0 conn(0x7fc2a80b1b60 legacy=0x7fc2a80ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:05.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.853+0000 7fc2a77fe640 1 -- 172.21.15.70:0/3644334306 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a80af920 con 0x7fc2a815ecb0 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.854+0000 7fc2a77fe640 1 --2- 172.21.15.70:0/3644334306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a815ecb0 0x7fc2a80b0b90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806a050 tx=0x7fc29802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581eb1eaadec0feb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.854+0000 7fc2a67fc640 1 -- 172.21.15.70:0/3644334306 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298034070 con 0x7fc2a815ecb0 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.854+0000 7fc2a67fc640 1 -- 172.21.15.70:0/3644334306 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc298030860 con 0x7fc2a815ecb0 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.854+0000 7fc2a67fc640 1 -- 172.21.15.70:0/3644334306 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298030c30 con 0x7fc2a815ecb0 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 -- 172.21.15.70:0/3644334306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a815ecb0 msgr2=0x7fc2a80b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/3644334306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a815ecb0 0x7fc2a80b0b90 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806a050 tx=0x7fc29802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 -- 172.21.15.70:0/3644334306 shutdown_connections 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/3644334306 >> v2:172.21.15.96:3300/0 conn(0x7fc2a80b4350 0x7fc2a80b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/3644334306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a815ecb0 0x7fc2a80b0b90 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 -- 172.21.15.70:0/3644334306 >> 172.21.15.70:0/3644334306 conn(0x7fc2a80b6870 msgr2=0x7fc2a80b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 -- 172.21.15.70:0/3644334306 shutdown_connections 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.855+0000 7fc2adaf2640 1 -- 172.21.15.70:0/3644334306 wait complete. 2024-04-15T15:16:05.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.856+0000 7fc2adaf2640 1 Processor -- start 2024-04-15T15:16:05.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.856+0000 7fc2adaf2640 1 -- start start 2024-04-15T15:16:05.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2a80b4350 0x7fc2a8161390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc2a815ecb0 0x7fc2a81618d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a80b47c0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a80afee0 con 0x7fc2a80b4350 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc2a815f6a0 con 0x7fc2a815ecb0 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41670/0 (socket says 172.21.15.70:41670) 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 -- 172.21.15.70:0/2538981865 learned_addr learned my addr 172.21.15.70:0/2538981865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 -- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc2a815ecb0 msgr2=0x7fc2a81618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc2a815ecb0 0x7fc2a81618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 -- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2a80b4350 msgr2=0x7fc2a8161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2a80b4350 0x7fc2a8161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 -- 172.21.15.70:0/2538981865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a80cd0e0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2a77fe640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc298000c00 tx=0x7fc2980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298034040 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc298039d90 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a80cd3c0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc298043610 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.857+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a80ca6d0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.858+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a80689e0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.859+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc29804a060 con 0x7fc2a80b1b60 2024-04-15T15:16:05.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.859+0000 7fc29ffff640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 0x7fc28806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:05.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.859+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc2980ae1a0 con 0x7fc2a80b1b60 2024-04-15T15:16:05.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.859+0000 7fc2a6ffd640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 0x7fc28806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:05.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.860+0000 7fc2a6ffd640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 0x7fc28806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81f7f90 tx=0x7fc290006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:05.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:05.864+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc29803e070 con 0x7fc2a80b1b60 2024-04-15T15:16:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.038+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc2a80ca9c0 con 0x7fc2a80b1b60 2024-04-15T15:16:06.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.038+0000 7fc29ffff640 1 -- 172.21.15.70:0/2538981865 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fc298037360 con 0x7fc2a80b1b60 2024-04-15T15:16:06.041 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.040+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 msgr2=0x7fc28806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.040+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 0x7fc28806edb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc2a81f7f90 tx=0x7fc290006c90 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 msgr2=0x7fc2a8154710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc298000c00 tx=0x7fc2980046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 shutdown_connections 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc2a815ecb0 0x7fc2a81618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc2a80b4350 0x7fc2a8161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc28806c8f0 0x7fc28806edb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 --2- 172.21.15.70:0/2538981865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc2a80b1b60 0x7fc2a8154710 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 >> 172.21.15.70:0/2538981865 conn(0x7fc2a80b6870 msgr2=0x7fc2a80b6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 shutdown_connections 2024-04-15T15:16:06.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.041+0000 7fc2adaf2640 1 -- 172.21.15.70:0/2538981865 wait complete. 2024-04-15T15:16:06.054 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.159+0000 7f25511c8640 1 Processor -- start 2024-04-15T15:16:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.159+0000 7f25511c8640 1 -- start start 2024-04-15T15:16:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f25511c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f254c15ecb0 0x7f254c0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f25511c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c0b1d30 0x7f254c0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f25511c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f254c0b10d0 con 0x7f254c0b4350 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f25511c8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f254c15f090 con 0x7f254c15ecb0 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f25511c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f254c0cdf80 con 0x7f254c0b1d30 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f254c15ecb0 0x7f254c0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f254c15ecb0 0x7f254c0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41672/0 (socket says 172.21.15.70:41672) 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 -- 172.21.15.70:0/913299718 learned_addr learned my addr 172.21.15.70:0/913299718 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254b59e640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c0b1d30 0x7f254c0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 -- 172.21.15.70:0/913299718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c0b1d30 msgr2=0x7f254c0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c0b1d30 0x7f254c0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 -- 172.21.15.70:0/913299718 >> v1:172.21.15.96:6789/0 conn(0x7f254c0b4350 legacy=0x7f254c0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.160+0000 7f254ad9d640 1 -- 172.21.15.70:0/913299718 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c0af920 con 0x7f254c15ecb0 2024-04-15T15:16:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.161+0000 7f254ad9d640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c15ecb0 0x7f254c0b0b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f253c009870 tx=0x7f253c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2ede410b493a07 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.161+0000 7f254a59c640 1 -- 172.21.15.70:0/913299718 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c02f840 con 0x7f254c15ecb0 2024-04-15T15:16:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.161+0000 7f254a59c640 1 -- 172.21.15.70:0/913299718 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c02f9e0 con 0x7f254c15ecb0 2024-04-15T15:16:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.161+0000 7f254a59c640 1 -- 172.21.15.70:0/913299718 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c02fdb0 con 0x7f254c15ecb0 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 -- 172.21.15.70:0/913299718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c15ecb0 msgr2=0x7f254c0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c15ecb0 0x7f254c0b0b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f253c009870 tx=0x7f253c02ba20 comp rx=0 tx=0).stop 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 -- 172.21.15.70:0/913299718 shutdown_connections 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c0b1d30 0x7f254c0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 --2- 172.21.15.70:0/913299718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c15ecb0 0x7f254c0b0b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 -- 172.21.15.70:0/913299718 >> 172.21.15.70:0/913299718 conn(0x7f254c0b6870 msgr2=0x7f254c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 -- 172.21.15.70:0/913299718 shutdown_connections 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.162+0000 7f25511c8640 1 -- 172.21.15.70:0/913299718 wait complete. 2024-04-15T15:16:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.163+0000 7f25511c8640 1 Processor -- start 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.163+0000 7f25511c8640 1 -- start start 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f254c0b4350 0x7f254c15fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c15ecb0 0x7f254c160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f254c0b47c0 con 0x7f254c0b1d30 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f254c0afee0 con 0x7f254c0b4350 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f254c160e40 con 0x7f254c15ecb0 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41684/0 (socket says 172.21.15.70:41684) 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 -- 172.21.15.70:0/2986689630 learned_addr learned my addr 172.21.15.70:0/2986689630 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254b59e640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c15ecb0 0x7f254c160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f2543fff640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f254c0b4350 0x7f254c15fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 -- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c15ecb0 msgr2=0x7f254c160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c15ecb0 0x7f254c160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 -- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f254c0b4350 msgr2=0x7f254c15fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f254c0b4350 0x7f254c15fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 -- 172.21.15.70:0/2986689630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254c161070 con 0x7f254c0b1d30 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f254ad9d640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f253c009990 tx=0x7f253c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c047070 con 0x7f254c0b1d30 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253c037d50 con 0x7f254c0b1d30 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253c0426a0 con 0x7f254c0b1d30 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.164+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f254c161350 con 0x7f254c0b1d30 2024-04-15T15:16:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.165+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f254c161910 con 0x7f254c0b1d30 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.166+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f254c0689e0 con 0x7f254c0b1d30 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.166+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f253c04a070 con 0x7f254c0b1d30 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.166+0000 7f25437fe640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 0x7f252c06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.166+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f253c0ae2f0 con 0x7f254c0b1d30 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.166+0000 7f2543fff640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 0x7f252c06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.167+0000 7f2543fff640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 0x7f252c06ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f254c0d1840 tx=0x7f2534009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.171+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f253c033030 con 0x7f254c0b1d30 2024-04-15T15:16:06.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.351+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f254c06c7c0 con 0x7f254c0b1d30 2024-04-15T15:16:06.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.351+0000 7f25437fe640 1 -- 172.21.15.70:0/2986689630 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f253c07bdf0 con 0x7f254c0b1d30 2024-04-15T15:16:06.353 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 msgr2=0x7f252c06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 0x7f252c06ede0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f254c0d1840 tx=0x7f2534009040 comp rx=0 tx=0).stop 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 msgr2=0x7f254c14b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f253c009990 tx=0x7f253c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 shutdown_connections 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f254c15ecb0 0x7f254c160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f254c0b4350 0x7f254c15fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f252c06c920 0x7f252c06ede0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 --2- 172.21.15.70:0/2986689630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f254c0b1d30 0x7f254c14b130 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 >> 172.21.15.70:0/2986689630 conn(0x7f254c0b6870 msgr2=0x7f254c0ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 shutdown_connections 2024-04-15T15:16:06.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.353+0000 7f25511c8640 1 -- 172.21.15.70:0/2986689630 wait complete. 2024-04-15T15:16:06.366 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:06.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.468+0000 7f5000641640 1 Processor -- start 2024-04-15T15:16:06.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.468+0000 7f5000641640 1 -- start start 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f5000641640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f5000641640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff80d1870 0x7f4ff8162a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f5000641640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ff80d0530 con 0x7f4ff80d13f0 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f5000641640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4ff80cf490 con 0x7f4ff815e7a0 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f5000641640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff80cf770 con 0x7f4ff80d1870 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41692/0 (socket says 172.21.15.70:41692) 2024-04-15T15:16:06.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 -- 172.21.15.70:0/839024845 learned_addr learned my addr 172.21.15.70:0/839024845 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 -- 172.21.15.70:0/839024845 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff80d1870 msgr2=0x7f4ff8162a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffebde640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff80d1870 0x7f4ff8162a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff80d1870 0x7f4ff8162a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 -- 172.21.15.70:0/839024845 >> v1:172.21.15.96:6789/0 conn(0x7f4ff80d13f0 legacy=0x7f4ff8162370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.469+0000 7f4ffe3dd640 1 -- 172.21.15.70:0/839024845 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff80cfa50 con 0x7f4ff815e7a0 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f4ffe3dd640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009870 tx=0x7f4ff402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe99c3b2c79b0209 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f4ffd3db640 1 -- 172.21.15.70:0/839024845 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff40317c0 con 0x7f4ff815e7a0 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f4ffd3db640 1 -- 172.21.15.70:0/839024845 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ff4037070 con 0x7f4ff815e7a0 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f4ffd3db640 1 -- 172.21.15.70:0/839024845 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4031b80 con 0x7f4ff815e7a0 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f5000641640 1 -- 172.21.15.70:0/839024845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff815e7a0 msgr2=0x7f4ff80d0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.470+0000 7f5000641640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009870 tx=0x7f4ff402b860 comp rx=0 tx=0).stop 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 -- 172.21.15.70:0/839024845 shutdown_connections 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff80d1870 0x7f4ff8162a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 --2- 172.21.15.70:0/839024845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff815e7a0 0x7f4ff80d0eb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 -- 172.21.15.70:0/839024845 >> 172.21.15.70:0/839024845 conn(0x7f4ff808bf60 msgr2=0x7f4ff80d4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 -- 172.21.15.70:0/839024845 shutdown_connections 2024-04-15T15:16:06.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.471+0000 7f5000641640 1 -- 172.21.15.70:0/839024845 wait complete. 2024-04-15T15:16:06.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.472+0000 7f5000641640 1 Processor -- start 2024-04-15T15:16:06.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.472+0000 7f5000641640 1 -- start start 2024-04-15T15:16:06.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80d13f0 0x7f4ff815a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff815e7a0 0x7f4ff80c7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff806db20 con 0x7f4ff80d1870 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ff80cf180 con 0x7f4ff80d13f0 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4ff815eb80 con 0x7f4ff815e7a0 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41694/0 (socket says 172.21.15.70:41694) 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 -- 172.21.15.70:0/1149794785 learned_addr learned my addr 172.21.15.70:0/1149794785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffebde640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff815e7a0 0x7f4ff80c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffe3dd640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80d13f0 0x7f4ff815a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 -- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff815e7a0 msgr2=0x7f4ff80c7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff815e7a0 0x7f4ff80c7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 -- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80d13f0 msgr2=0x7f4ff815a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80d13f0 0x7f4ff815a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 -- 172.21.15.70:0/1149794785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ff814b810 con 0x7f4ff80d1870 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4ffdbdc640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b410 tx=0x7f4fe800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8015020 con 0x7f4ff80d1870 2024-04-15T15:16:06.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4fe8011830 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ff814baf0 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fe8011c20 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.473+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ff80c7690 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.474+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ff80689e0 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.475+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4fe8018020 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.475+0000 7f4feeffd640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 0x7f4fdc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.475+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4fe808bed0 con 0x7f4ff80d1870 2024-04-15T15:16:06.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.475+0000 7f4ffe3dd640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 0x7f4fdc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.476+0000 7f4ffe3dd640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 0x7f4fdc06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4036000 tx=0x7f4ff403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.480+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4fe801f020 con 0x7f4ff80d1870 2024-04-15T15:16:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.665+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4ff80d1d60 con 0x7f4ff80d1870 2024-04-15T15:16:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.665+0000 7f4feeffd640 1 -- 172.21.15.70:0/1149794785 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f4fe80172c0 con 0x7f4ff80d1870 2024-04-15T15:16:06.667 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:06.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 msgr2=0x7f4fdc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 0x7f4fdc06edb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4036000 tx=0x7f4ff403d040 comp rx=0 tx=0).stop 2024-04-15T15:16:06.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 msgr2=0x7f4ff814b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f4fe800b410 tx=0x7f4fe800b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 shutdown_connections 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ff815e7a0 0x7f4ff80c7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4fdc06c8f0 0x7f4fdc06edb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ff80d1870 0x7f4ff814b0f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 --2- 172.21.15.70:0/1149794785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ff80d13f0 0x7f4ff815a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 >> 172.21.15.70:0/1149794785 conn(0x7f4ff808bf60 msgr2=0x7f4ff80d3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 shutdown_connections 2024-04-15T15:16:06.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.667+0000 7f5000641640 1 -- 172.21.15.70:0/1149794785 wait complete. 2024-04-15T15:16:06.680 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:16:06.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:06.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:06.709 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:06.709 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:16:06.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:06.709 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:16:06.767 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:06.767 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:06.787 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:06.787 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:06.846 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:06.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.947+0000 7ff0e0ab2640 1 Processor -- start 2024-04-15T15:16:06.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.947+0000 7ff0e0ab2640 1 -- start start 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.948+0000 7ff0e0ab2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.948+0000 7ff0e0ab2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 0x7ff0dc072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.948+0000 7ff0e0ab2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0dc15c1d0 con 0x7ff0dc072c60 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.948+0000 7ff0e0ab2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff0dc0b4da0 con 0x7ff0dc0b5230 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.948+0000 7ff0e0ab2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff0dc0b4190 con 0x7ff0dc158770 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0d9d9b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 0x7ff0dc072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42088/0 (socket says 172.21.15.70:42088) 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 -- 172.21.15.70:0/696630770 learned_addr learned my addr 172.21.15.70:0/696630770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0dad9d640 1 --1- 172.21.15.70:0/696630770 >> v1:172.21.15.96:6789/0 conn(0x7ff0dc072c60 0x7ff0dc15e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:45028/0 (socket says 172.21.15.70:45028) 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 -- 172.21.15.70:0/696630770 >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 msgr2=0x7ff0dc072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:06.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 --2- 172.21.15.70:0/696630770 >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 0x7ff0dc072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 -- 172.21.15.70:0/696630770 >> v1:172.21.15.96:6789/0 conn(0x7ff0dc072c60 legacy=0x7ff0dc15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0da59c640 1 -- 172.21.15.70:0/696630770 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc0b4470 con 0x7ff0dc158770 2024-04-15T15:16:06.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.949+0000 7ff0d9d9b640 1 --2- 172.21.15.70:0/696630770 >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 0x7ff0dc072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.950+0000 7ff0da59c640 1 --2- 172.21.15.70:0/696630770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009870 tx=0x7ff0cc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6038b418d48e4386 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.950+0000 7ff0d959a640 1 -- 172.21.15.70:0/696630770 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc034070 con 0x7ff0dc158770 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.950+0000 7ff0d959a640 1 -- 172.21.15.70:0/696630770 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc030840 con 0x7ff0dc158770 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.950+0000 7ff0d959a640 1 -- 172.21.15.70:0/696630770 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc030c10 con 0x7ff0dc158770 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.950+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/696630770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 msgr2=0x7ff0dc15bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/696630770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc009870 tx=0x7ff0cc02b980 comp rx=0 tx=0).stop 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/696630770 shutdown_connections 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/696630770 >> v2:172.21.15.96:3300/0 conn(0x7ff0dc0b5230 0x7ff0dc072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/696630770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc158770 0x7ff0dc15bc90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/696630770 >> 172.21.15.70:0/696630770 conn(0x7ff0dc06be00 msgr2=0x7ff0dc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/696630770 shutdown_connections 2024-04-15T15:16:06.954 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.951+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/696630770 wait complete. 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.952+0000 7ff0e0ab2640 1 Processor -- start 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.952+0000 7ff0e0ab2640 1 -- start start 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc0b5230 0x7ff0dc15d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 0x7ff0dc15dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc071fa0 con 0x7ff0dc072c60 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff0dc0b4a30 con 0x7ff0dc158770 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0e0ab2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff0dc158b50 con 0x7ff0dc0b5230 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41702/0 (socket says 172.21.15.70:41702) 2024-04-15T15:16:06.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0dad9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 0x7ff0dc15dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 -- 172.21.15.70:0/1275508438 learned_addr learned my addr 172.21.15.70:0/1275508438 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0d9d9b640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc0b5230 0x7ff0dc15d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 -- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc0b5230 msgr2=0x7ff0dc15d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc0b5230 0x7ff0dc15d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 -- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 msgr2=0x7ff0dc15dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 0x7ff0dc15dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 -- 172.21.15.70:0/1275508438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0dc14d0d0 con 0x7ff0dc072c60 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0dad9d640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 0x7ff0dc15dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.953+0000 7ff0da59c640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.954+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc034040 con 0x7ff0dc072c60 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.954+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0cc039d90 con 0x7ff0dc072c60 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.954+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc043630 con 0x7ff0dc072c60 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.954+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0dc0cac10 con 0x7ff0dc072c60 2024-04-15T15:16:06.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.954+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0dc0cb1d0 con 0x7ff0dc072c60 2024-04-15T15:16:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.955+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0dc0689e0 con 0x7ff0dc072c60 2024-04-15T15:16:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.955+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff0cc04a060 con 0x7ff0dc072c60 2024-04-15T15:16:06.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.956+0000 7ff0caffd640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.956+0000 7ff0d9d9b640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.956+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff0cc0addb0 con 0x7ff0dc072c60 2024-04-15T15:16:06.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.956+0000 7ff0d9d9b640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc1f9350 tx=0x7ff0d000a2a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:06.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:06.960+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff0dc14d0d0 con 0x7ff0dc072c60 2024-04-15T15:16:07.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.136+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0dc0af6b0 con 0x7ff0dc072c60 2024-04-15T15:16:07.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.137+0000 7ff0caffd640 1 -- 172.21.15.70:0/1275508438 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7ff0cc03e070 con 0x7ff0dc072c60 2024-04-15T15:16:07.139 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:07.139 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 msgr2=0x7ff0bc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc1f9350 tx=0x7ff0d000a2a0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 msgr2=0x7ff0dc14cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc000c00 tx=0x7ff0cc0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 shutdown_connections 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff0dc158770 0x7ff0dc15dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff0dc0b5230 0x7ff0dc15d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff0bc06c8f0 0x7ff0bc06edb0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 --2- 172.21.15.70:0/1275508438 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0dc072c60 0x7ff0dc14cb90 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.139+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 >> 172.21.15.70:0/1275508438 conn(0x7ff0dc06be00 msgr2=0x7ff0dc0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.140+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 shutdown_connections 2024-04-15T15:16:07.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.140+0000 7ff0e0ab2640 1 -- 172.21.15.70:0/1275508438 wait complete. 2024-04-15T15:16:07.153 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:07.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.258+0000 7fc75b6d3640 1 Processor -- start 2024-04-15T15:16:07.260 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.258+0000 7fc75b6d3640 1 -- start start 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75b6d3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc75409a9d0 0x7fc754160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75b6d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 0x7fc75415fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75b6d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc754070720 con 0x7fc75409adb0 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75b6d3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc75409ba90 con 0x7fc75409a9d0 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75b6d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc754160070 con 0x7fc7540701f0 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc75409a9d0 0x7fc754160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc75409a9d0 0x7fc754160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41712/0 (socket says 172.21.15.70:41712) 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 -- 172.21.15.70:0/23234643 learned_addr learned my addr 172.21.15.70:0/23234643 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 -- 172.21.15.70:0/23234643 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 msgr2=0x7fc75415fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.262 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc758c6e640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 0x7fc75415fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 0x7fc75415fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 -- 172.21.15.70:0/23234643 >> v1:172.21.15.96:6789/0 conn(0x7fc75409adb0 legacy=0x7fc7540d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc75946f640 1 -- 172.21.15.70:0/23234643 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754070a90 con 0x7fc75409a9d0 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.259+0000 7fc758c6e640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 0x7fc75415fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.260+0000 7fc75946f640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc75409a9d0 0x7fc754160420 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc7500098a0 tx=0x7fc75002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b0ab1b56a07ee6a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.260+0000 7fc74bfff640 1 -- 172.21.15.70:0/23234643 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750035020 con 0x7fc75409a9d0 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.260+0000 7fc74bfff640 1 -- 172.21.15.70:0/23234643 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc750031640 con 0x7fc75409a9d0 2024-04-15T15:16:07.263 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.260+0000 7fc74bfff640 1 -- 172.21.15.70:0/23234643 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75002d040 con 0x7fc75409a9d0 2024-04-15T15:16:07.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 -- 172.21.15.70:0/23234643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc75409a9d0 msgr2=0x7fc754160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc75409a9d0 0x7fc754160420 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fc7500098a0 tx=0x7fc75002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 -- 172.21.15.70:0/23234643 shutdown_connections 2024-04-15T15:16:07.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc7540701f0 0x7fc75415fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/23234643 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc75409a9d0 0x7fc754160420 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 -- 172.21.15.70:0/23234643 >> 172.21.15.70:0/23234643 conn(0x7fc7540d2460 msgr2=0x7fc7540d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 -- 172.21.15.70:0/23234643 shutdown_connections 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.261+0000 7fc75b6d3640 1 -- 172.21.15.70:0/23234643 wait complete. 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.262+0000 7fc75b6d3640 1 Processor -- start 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.262+0000 7fc75b6d3640 1 -- start start 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 0x7fc7541534c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 0x7fc7540ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc754071050 con 0x7fc7540701f0 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc7540d08c0 con 0x7fc75409adb0 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc7540b1c60 con 0x7fc75409a9d0 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41718/0 (socket says 172.21.15.70:41718) 2024-04-15T15:16:07.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 -- 172.21.15.70:0/2907796702 learned_addr learned my addr 172.21.15.70:0/2907796702 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 -- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 msgr2=0x7fc7541534c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc758c6e640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 0x7fc7541534c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc759c70640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 0x7fc7540ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 0x7fc7541534c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 -- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 msgr2=0x7fc7540ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 0x7fc7540ce6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 -- 172.21.15.70:0/2907796702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754153a00 con 0x7fc7540701f0 2024-04-15T15:16:07.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc759c70640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 0x7fc7540ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc758c6e640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 0x7fc7541534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75946f640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc7500097c0 tx=0x7fc7500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750040070 con 0x7fc7540701f0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc750039cb0 con 0x7fc7540701f0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc754161790 con 0x7fc7540701f0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.263+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc750035040 con 0x7fc7540701f0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.264+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc754160370 con 0x7fc7540701f0 2024-04-15T15:16:07.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.265+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc754161cb0 con 0x7fc7540701f0 2024-04-15T15:16:07.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.265+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc75004b060 con 0x7fc7540701f0 2024-04-15T15:16:07.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.265+0000 7fc749ffb640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 0x7fc73806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.266+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc7500ad9c0 con 0x7fc7540701f0 2024-04-15T15:16:07.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.266+0000 7fc758c6e640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 0x7fc73806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.266+0000 7fc758c6e640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 0x7fc73806ede0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc744006fd0 tx=0x7fc744008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.270+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc7500afe20 con 0x7fc7540701f0 2024-04-15T15:16:07.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.447+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc7540689e0 con 0x7fc7540701f0 2024-04-15T15:16:07.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.447+0000 7fc749ffb640 1 -- 172.21.15.70:0/2907796702 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc750039e60 con 0x7fc7540701f0 2024-04-15T15:16:07.449 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 msgr2=0x7fc73806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 0x7fc73806ede0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc744006fd0 tx=0x7fc744008040 comp rx=0 tx=0).stop 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 msgr2=0x7fc7540c5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc7500097c0 tx=0x7fc7500046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 shutdown_connections 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc75409adb0 0x7fc7540ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc75409a9d0 0x7fc7541534c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc73806c920 0x7fc73806ede0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 --2- 172.21.15.70:0/2907796702 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc7540701f0 0x7fc7540c5f90 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 >> 172.21.15.70:0/2907796702 conn(0x7fc7540d2460 msgr2=0x7fc75406cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 shutdown_connections 2024-04-15T15:16:07.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.449+0000 7fc75b6d3640 1 -- 172.21.15.70:0/2907796702 wait complete. 2024-04-15T15:16:07.462 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:07.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.567+0000 7fdfc19ba640 1 Processor -- start 2024-04-15T15:16:07.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.567+0000 7fdfc19ba640 1 -- start start 2024-04-15T15:16:07.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfc19ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc072d80 0x7fdfbc071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfc19ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdfbc0d0da0 0x7fdfbc0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfc19ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfbc15db40 con 0x7fdfbc0af3e0 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfc19ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fdfbc15cfc0 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfc19ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdfbc15d2a0 con 0x7fdfbc072d80 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdfbc0d0da0 0x7fdfbc0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fdfbc0d0da0 0x7fdfbc0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41726/0 (socket says 172.21.15.70:41726) 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/503064262 learned_addr learned my addr 172.21.15.70:0/503064262 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbaffd640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc072d80 0x7fdfbc071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/503064262 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc072d80 msgr2=0x7fdfbc071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc072d80 0x7fdfbc071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/503064262 >> v1:172.21.15.96:6789/0 conn(0x7fdfbc0af3e0 legacy=0x7fdfbc0d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.568+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/503064262 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfbc15d580 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.569+0000 7fdfbb7fe640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc0af840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009e50 tx=0x7fdfb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db36dcaa70ca7a22 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.569+0000 7fdfba7fc640 1 -- 172.21.15.70:0/503064262 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb402ca70 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.569+0000 7fdfba7fc640 1 -- 172.21.15.70:0/503064262 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb402cc10 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.569+0000 7fdfba7fc640 1 -- 172.21.15.70:0/503064262 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb40343f0 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 -- 172.21.15.70:0/503064262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 msgr2=0x7fdfbc0af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc0af840 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4009e50 tx=0x7fdfb402bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 -- 172.21.15.70:0/503064262 shutdown_connections 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc0af840 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/503064262 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc072d80 0x7fdfbc071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 -- 172.21.15.70:0/503064262 >> 172.21.15.70:0/503064262 conn(0x7fdfbc06be00 msgr2=0x7fdfbc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 -- 172.21.15.70:0/503064262 shutdown_connections 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.570+0000 7fdfc19ba640 1 -- 172.21.15.70:0/503064262 wait complete. 2024-04-15T15:16:07.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.571+0000 7fdfc19ba640 1 Processor -- start 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.571+0000 7fdfc19ba640 1 -- start start 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfbc072d80 0x7fdfbc0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc0af3e0 0x7fdfbc0cf030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc15cd10 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdfbc15cb90 con 0x7fdfbc072d80 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfc19ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdfbc06e060 con 0x7fdfbc0af3e0 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41730/0 (socket says 172.21.15.70:41730) 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/514718935 learned_addr learned my addr 172.21.15.70:0/514718935 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbaffd640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfbc072d80 0x7fdfbc0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfb27fc640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc0af3e0 0x7fdfbc0cf030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc0af3e0 msgr2=0x7fdfbc0cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc0af3e0 0x7fdfbc0cf030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfbc072d80 msgr2=0x7fdfbc0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfbc072d80 0x7fdfbc0cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.572+0000 7fdfbb7fe640 1 -- 172.21.15.70:0/514718935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfbc0ce080 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfbb7fe640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4036040 tx=0x7fdfb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4043020 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfb403e450 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4009510 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfbc157460 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.573+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfbc157a20 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.574+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfbc0689e0 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.574+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdfb4046020 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.575+0000 7fdfb8ff9640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 0x7fdf9c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.575+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdfb40ae630 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.575+0000 7fdfbaffd640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 0x7fdf9c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.575+0000 7fdfbaffd640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 0x7fdf9c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009b40 tx=0x7fdfac009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.579+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdfb404d030 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.754+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfbc072500 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.754+0000 7fdfb8ff9640 1 -- 172.21.15.70:0/514718935 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fdfb407c010 con 0x7fdfbc0d0da0 2024-04-15T15:16:07.756 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 msgr2=0x7fdf9c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 0x7fdf9c06edb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fdfac009b40 tx=0x7fdfac009190 comp rx=0 tx=0).stop 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 msgr2=0x7fdfbc152250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fdfb4036040 tx=0x7fdfb4004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 shutdown_connections 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdf9c06c8f0 0x7fdf9c06edb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdfbc0d0da0 0x7fdfbc152250 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdfbc0af3e0 0x7fdfbc0cf030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 --2- 172.21.15.70:0/514718935 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdfbc072d80 0x7fdfbc0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 >> 172.21.15.70:0/514718935 conn(0x7fdfbc06be00 msgr2=0x7fdfbc15bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 shutdown_connections 2024-04-15T15:16:07.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.756+0000 7fdfc19ba640 1 -- 172.21.15.70:0/514718935 wait complete. 2024-04-15T15:16:07.769 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:07.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 Processor -- start 2024-04-15T15:16:07.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 -- start start 2024-04-15T15:16:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd42c09ac40 0x7fd42c0d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd42c0b49d0 0x7fd42c0ce030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c0b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd42c0b1b70 con 0x7fd42c0b49d0 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd42c0d1210 con 0x7fd42c09ac40 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.874+0000 7fd43200d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c0d14b0 con 0x7fd42c0ce570 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd42c0b49d0 0x7fd42c0ce030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd42c0b49d0 0x7fd42c0ce030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41744/0 (socket says 172.21.15.70:41744) 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 -- 172.21.15.70:0/4210602292 learned_addr learned my addr 172.21.15.70:0/4210602292 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd42b7fe640 1 --2- 172.21.15.70:0/4210602292 >> v2:172.21.15.96:3300/0 conn(0x7fd42c09ac40 0x7fd42c0d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd42bfff640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c0b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 -- 172.21.15.70:0/4210602292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 msgr2=0x7fd42c0b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c0b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 -- 172.21.15.70:0/4210602292 >> v2:172.21.15.96:3300/0 conn(0x7fd42c09ac40 msgr2=0x7fd42c0d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:07.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 --2- 172.21.15.70:0/4210602292 >> v2:172.21.15.96:3300/0 conn(0x7fd42c09ac40 0x7fd42c0d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.875+0000 7fd422ffd640 1 -- 172.21.15.70:0/4210602292 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c0cd550 con 0x7fd42c0b49d0 2024-04-15T15:16:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd422ffd640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c0b49d0 0x7fd42c0ce030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd42c1613d0 tx=0x7fd41402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8be1e7010c54e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd42affd640 1 -- 172.21.15.70:0/4210602292 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414034070 con 0x7fd42c0b49d0 2024-04-15T15:16:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd42affd640 1 -- 172.21.15.70:0/4210602292 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd414030860 con 0x7fd42c0b49d0 2024-04-15T15:16:07.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd42affd640 1 -- 172.21.15.70:0/4210602292 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414030c30 con 0x7fd42c0b49d0 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 -- 172.21.15.70:0/4210602292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c0b49d0 msgr2=0x7fd42c0ce030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c0b49d0 0x7fd42c0ce030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd42c1613d0 tx=0x7fd41402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 -- 172.21.15.70:0/4210602292 shutdown_connections 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c0b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 --2- 172.21.15.70:0/4210602292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c0b49d0 0x7fd42c0ce030 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 --2- 172.21.15.70:0/4210602292 >> v2:172.21.15.96:3300/0 conn(0x7fd42c09ac40 0x7fd42c0d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 -- 172.21.15.70:0/4210602292 >> 172.21.15.70:0/4210602292 conn(0x7fd42c0b3eb0 msgr2=0x7fd42c0b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 -- 172.21.15.70:0/4210602292 shutdown_connections 2024-04-15T15:16:07.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.876+0000 7fd43200d640 1 -- 172.21.15.70:0/4210602292 wait complete. 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 Processor -- start 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 -- start start 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 0x7fd42c1e60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c1e6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c09b020 con 0x7fd42c09ac40 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd42c0cd1e0 con 0x7fd42c0b49d0 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.878+0000 7fd43200d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd42c09b1a0 con 0x7fd42c0ce570 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41748/0 (socket says 172.21.15.70:41748) 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 -- 172.21.15.70:0/3134003467 learned_addr learned my addr 172.21.15.70:0/3134003467 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42bfff640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c1e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd422ffd640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 0x7fd42c1e60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 -- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 msgr2=0x7fd42c1e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c1e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 -- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 msgr2=0x7fd42c1e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 0x7fd42c1e60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 -- 172.21.15.70:0/3134003467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c1e7420 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd422ffd640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 0x7fd42c1e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd42b7fe640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00e890 tx=0x7fd41c00ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c017070 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41c00c940 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd42c0b8060 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41c00cd10 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.879+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd42c0cbb10 con 0x7fd42c09ac40 2024-04-15T15:16:07.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.880+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd42c0689e0 con 0x7fd42c09ac40 2024-04-15T15:16:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.881+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd41c01a070 con 0x7fd42c09ac40 2024-04-15T15:16:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.881+0000 7fd428ff9640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.881+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd41c08be20 con 0x7fd42c09ac40 2024-04-15T15:16:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.881+0000 7fd422ffd640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:07.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.882+0000 7fd422ffd640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd414036000 tx=0x7fd41403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:07.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:07.885+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd42c1e7420 con 0x7fd42c09ac40 2024-04-15T15:16:08.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.059+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd42c0c9150 con 0x7fd42c09ac40 2024-04-15T15:16:08.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.059+0000 7fd428ff9640 1 -- 172.21.15.70:0/3134003467 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fd41c059890 con 0x7fd42c09ac40 2024-04-15T15:16:08.061 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:08.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 msgr2=0x7fd40c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 0x7fd40c06edb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd414036000 tx=0x7fd41403b040 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 msgr2=0x7fd42c0c8830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd41c00e890 tx=0x7fd41c00ed50 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 shutdown_connections 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd42c0ce570 0x7fd42c1e6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd42c0b49d0 0x7fd42c1e60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd40c06c8f0 0x7fd40c06edb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 --2- 172.21.15.70:0/3134003467 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd42c09ac40 0x7fd42c0c8830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 >> 172.21.15.70:0/3134003467 conn(0x7fd42c0b3eb0 msgr2=0x7fd42c0b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 shutdown_connections 2024-04-15T15:16:08.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.061+0000 7fd43200d640 1 -- 172.21.15.70:0/3134003467 wait complete. 2024-04-15T15:16:08.075 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:08.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.179+0000 7fdb10909640 1 Processor -- start 2024-04-15T15:16:08.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.179+0000 7fdb10909640 1 -- start start 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb10909640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 0x7fdb0c160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb10909640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0c06f090 con 0x7fdb0c15d090 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb10909640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb0c15d660 con 0x7fdb0c067720 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb10909640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c160130 con 0x7fdb0c06ec00 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb0a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdb0c15d090 0x7fdb0c06e4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51374/0 (socket says 172.21.15.70:51374) 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb0a59c640 1 -- 172.21.15.70:0/2274747810 learned_addr learned my addr 172.21.15.70:0/2274747810 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.180+0000 7fdb0ad9d640 1 --2- 172.21.15.70:0/2274747810 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 0x7fdb0c160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 570039257 0 0) 0x7fdb0c06f090 con 0x7fdb0c15d090 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdaf4003520 con 0x7fdb0c15d090 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2320324136 0 0) 0x7fdb0c15d660 con 0x7fdb0c067720 2024-04-15T15:16:08.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb0c06f090 con 0x7fdb0c067720 2024-04-15T15:16:08.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0ad9d640 1 -- 172.21.15.70:0/2274747810 >> v1:172.21.15.96:6789/0 conn(0x7fdb0c067720 legacy=0x7fdb0c0ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0ad9d640 1 -- 172.21.15.70:0/2274747810 >> v1:172.21.15.70:6789/0 conn(0x7fdb0c15d090 legacy=0x7fdb0c06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.181+0000 7fdb0ad9d640 1 -- 172.21.15.70:0/2274747810 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c15d940 con 0x7fdb0c06ec00 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb0ad9d640 1 --2- 172.21.15.70:0/2274747810 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 0x7fdb0c160b80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009dc0 tx=0x7fdb0402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ea128af12a4d9d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04039070 con 0x7fdb0c06ec00 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb0402cc40 con 0x7fdb0c06ec00 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb0959a640 1 -- 172.21.15.70:0/2274747810 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb04034440 con 0x7fdb0c06ec00 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 -- 172.21.15.70:0/2274747810 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 msgr2=0x7fdb0c160b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 --2- 172.21.15.70:0/2274747810 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 0x7fdb0c160b80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009dc0 tx=0x7fdb0402be90 comp rx=0 tx=0).stop 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 -- 172.21.15.70:0/2274747810 shutdown_connections 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 --2- 172.21.15.70:0/2274747810 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c06ec00 0x7fdb0c160b80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 -- 172.21.15.70:0/2274747810 >> 172.21.15.70:0/2274747810 conn(0x7fdb0c073360 msgr2=0x7fdb0c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 -- 172.21.15.70:0/2274747810 shutdown_connections 2024-04-15T15:16:08.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.182+0000 7fdb10909640 1 -- 172.21.15.70:0/2274747810 wait complete. 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 Processor -- start 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 -- start start 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb0c067720 0x7fdb0c156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c15d090 0x7fdb0c0aa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c160660 con 0x7fdb0c06ec00 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fdb0c1610c0 con 0x7fdb0c067720 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.184+0000 7fdb10909640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fdb0c0b0460 con 0x7fdb0c15d090 2024-04-15T15:16:08.188 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41762/0 (socket says 172.21.15.70:41762) 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 -- 172.21.15.70:0/462588655 learned_addr learned my addr 172.21.15.70:0/462588655 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb0ad9d640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c15d090 0x7fdb0c0aa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb0a59c640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb0c067720 0x7fdb0c156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 -- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c15d090 msgr2=0x7fdb0c0aa9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c15d090 0x7fdb0c0aa9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 -- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb0c067720 msgr2=0x7fdb0c156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb0c067720 0x7fdb0c156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 -- 172.21.15.70:0/462588655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb0c1f2fd0 con 0x7fdb0c06ec00 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb09d9b640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000c430 tx=0x7fdb0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb00017070 con 0x7fdb0c06ec00 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb000052f0 con 0x7fdb0c06ec00 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb0c0a9950 con 0x7fdb0c06ec00 2024-04-15T15:16:08.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb00003830 con 0x7fdb0c06ec00 2024-04-15T15:16:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.185+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb0c0a9f10 con 0x7fdb0c06ec00 2024-04-15T15:16:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.186+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdb0c0689e0 con 0x7fdb0c06ec00 2024-04-15T15:16:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.187+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fdb0001a070 con 0x7fdb0c06ec00 2024-04-15T15:16:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.187+0000 7fdafaffd640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 0x7fdaec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.188+0000 7fdb0a59c640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 0x7fdaec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.188+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdb0008b280 con 0x7fdb0c06ec00 2024-04-15T15:16:08.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.188+0000 7fdb0a59c640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 0x7fdaec06edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0011e0 tx=0x7fdafc007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.192+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fdb0c1f2fd0 con 0x7fdb0c06ec00 2024-04-15T15:16:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.367+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fdb0c06d520 con 0x7fdb0c06ec00 2024-04-15T15:16:08.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.367+0000 7fdafaffd640 1 -- 172.21.15.70:0/462588655 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fdb00058c60 con 0x7fdb0c06ec00 2024-04-15T15:16:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 msgr2=0x7fdaec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 0x7fdaec06edb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0011e0 tx=0x7fdafc007d30 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 msgr2=0x7fdb0c1f2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdb0000c430 tx=0x7fdb0000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 shutdown_connections 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fdb0c15d090 0x7fdb0c0aa9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fdaec06c8f0 0x7fdaec06edb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fdb0c06ec00 0x7fdb0c1f2a90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 --2- 172.21.15.70:0/462588655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fdb0c067720 0x7fdb0c156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 >> 172.21.15.70:0/462588655 conn(0x7fdb0c073360 msgr2=0x7fdb0c074df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 shutdown_connections 2024-04-15T15:16:08.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.369+0000 7fdb10909640 1 -- 172.21.15.70:0/462588655 wait complete. 2024-04-15T15:16:08.382 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:08.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.484+0000 7faff83f4640 1 Processor -- start 2024-04-15T15:16:08.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.484+0000 7faff83f4640 1 -- start start 2024-04-15T15:16:08.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff83f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff015d090 0x7faff0160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff83f4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faff0067720 0x7faff006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff83f4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faff015d660 con 0x7faff006ea30 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff83f4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7faff0160130 con 0x7faff0067720 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff83f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faff01610c0 con 0x7faff015d090 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff598f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faff0067720 0x7faff006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff598f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7faff0067720 0x7faff006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41778/0 (socket says 172.21.15.70:41778) 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff598f640 1 -- 172.21.15.70:0/17211286 learned_addr learned my addr 172.21.15.70:0/17211286 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.485+0000 7faff6190640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff015d090 0x7faff0160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.486+0000 7faff598f640 1 -- 172.21.15.70:0/17211286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff015d090 msgr2=0x7faff0160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.486+0000 7faff598f640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff015d090 0x7faff0160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.486+0000 7faff598f640 1 -- 172.21.15.70:0/17211286 >> v1:172.21.15.96:6789/0 conn(0x7faff006ea30 legacy=0x7faff00cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.486+0000 7faff598f640 1 -- 172.21.15.70:0/17211286 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff015d940 con 0x7faff0067720 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.486+0000 7faff598f640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff0067720 0x7faff006e4f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009870 tx=0x7fafe002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=726b4707785d1c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff518e640 1 -- 172.21.15.70:0/17211286 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0035020 con 0x7faff0067720 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff518e640 1 -- 172.21.15.70:0/17211286 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe0031640 con 0x7faff0067720 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff518e640 1 -- 172.21.15.70:0/17211286 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe002d040 con 0x7faff0067720 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 -- 172.21.15.70:0/17211286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff0067720 msgr2=0x7faff006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff0067720 0x7faff006e4f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fafe0009870 tx=0x7fafe002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 -- 172.21.15.70:0/17211286 shutdown_connections 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff0067720 0x7faff006e4f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 --2- 172.21.15.70:0/17211286 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff015d090 0x7faff0160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 -- 172.21.15.70:0/17211286 >> 172.21.15.70:0/17211286 conn(0x7faff0073360 msgr2=0x7faff006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 -- 172.21.15.70:0/17211286 shutdown_connections 2024-04-15T15:16:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.487+0000 7faff83f4640 1 -- 172.21.15.70:0/17211286 wait complete. 2024-04-15T15:16:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.488+0000 7faff83f4640 1 Processor -- start 2024-04-15T15:16:08.491 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.488+0000 7faff83f4640 1 -- start start 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff0067720 0x7faff015b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 0x7faff0157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faff00b0460 con 0x7faff006ea30 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7faff0160660 con 0x7faff015d090 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff83f4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7faff00b05e0 con 0x7faff0067720 2024-04-15T15:16:08.495 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41794/0 (socket says 172.21.15.70:41794) 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 -- 172.21.15.70:0/1128275582 learned_addr learned my addr 172.21.15.70:0/1128275582 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff6190640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff0067720 0x7faff015b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff6991640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 0x7faff0157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 -- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff0067720 msgr2=0x7faff015b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff0067720 0x7faff015b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 -- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 msgr2=0x7faff0157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 0x7faff0157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff598f640 1 -- 172.21.15.70:0/1128275582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faff0152890 con 0x7faff006ea30 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.489+0000 7faff6991640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 0x7faff0157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7faff598f640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fafe0000c00 tx=0x7fafe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0040070 con 0x7faff006ea30 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafe00399f0 con 0x7faff006ea30 2024-04-15T15:16:08.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafe0035040 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faff0152bd0 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.490+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faff0153410 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.491+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faff00689e0 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.491+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fafe004b060 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.492+0000 7fafe6ffd640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 0x7fafd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.492+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fafe00ad7c0 con 0x7faff006ea30 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.492+0000 7faff6190640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 0x7fafd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.492+0000 7faff6190640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 0x7fafd406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fafec011340 tx=0x7fafec00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.498 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.496+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7faff0152bd0 con 0x7faff006ea30 2024-04-15T15:16:08.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.683+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7faff006e810 con 0x7faff006ea30 2024-04-15T15:16:08.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.684+0000 7fafe6ffd640 1 -- 172.21.15.70:0/1128275582 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fafe007b760 con 0x7faff006ea30 2024-04-15T15:16:08.687 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 msgr2=0x7fafd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 0x7fafd406edb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fafec011340 tx=0x7fafec00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 msgr2=0x7faff0153930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fafe0000c00 tx=0x7fafe00046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 shutdown_connections 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7faff015d090 0x7faff0157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fafd406c8f0 0x7fafd406edb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faff006ea30 0x7faff0153930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 --2- 172.21.15.70:0/1128275582 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7faff0067720 0x7faff015b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 >> 172.21.15.70:0/1128275582 conn(0x7faff0073360 msgr2=0x7faff00b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 shutdown_connections 2024-04-15T15:16:08.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.686+0000 7faff83f4640 1 -- 172.21.15.70:0/1128275582 wait complete. 2024-04-15T15:16:08.699 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:08.699 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:08.719 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:08.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.821+0000 7f54e957c640 1 Processor -- start 2024-04-15T15:16:08.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.821+0000 7f54e957c640 1 -- start start 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e957c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54e40b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e957c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54e40b47a0 con 0x7f54e40d2360 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e957c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54e40b4a80 con 0x7f54e40d3270 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e957c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f54e40b53b0 con 0x7f54e415e700 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54dbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f54e40d2360 0x7f54e40d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51376/0 (socket says 172.21.15.70:51376) 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54dbfff640 1 -- 172.21.15.70:0/250103437 learned_addr learned my addr 172.21.15.70:0/250103437 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2515504447 0 0) 0x7f54e40b47a0 con 0x7f54e40d2360 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54e40b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54d0003520 con 0x7f54e40d2360 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 32807687 0 0) 0x7f54d0003520 con 0x7f54e40d2360 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.822+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 msgr2=0x7f54e40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54e40b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 >> v1:172.21.15.96:6789/0 conn(0x7f54e40d3270 legacy=0x7f54e40cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e40b4d60 con 0x7f54e40d2360 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f54cc001670 con 0x7f54e40d2360 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 >> v1:172.21.15.70:6789/0 conn(0x7f54e40d2360 legacy=0x7f54e40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d00039e0 0x7f54d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54d0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d0004d40 0x7f54d00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54e40b53b0 con 0x7f54d0004d40 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54d000b9c0 con 0x7f54d00039e0 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f54d000bbc0 con 0x7f54e415e700 2024-04-15T15:16:08.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d00039e0 0x7f54d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54dbfff640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54d0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e37fe640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d0004d40 0x7f54d00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 -- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 msgr2=0x7f54d0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54d0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 -- 172.21.15.70:0/250103437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d0004d40 msgr2=0x7f54d00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d0004d40 0x7f54d00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e2ffd640 1 -- 172.21.15.70:0/250103437 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d0003520 con 0x7f54d00039e0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54dbfff640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54e415e700 0x7f54d0004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.823+0000 7f54e37fe640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d0004d40 0x7f54d00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.824+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d00039e0 0x7f54d0003dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009ab0 tx=0x7f54d0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4094c602cc9f67 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.824+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4009680 con 0x7f54d00039e0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.824+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54d4005990 con 0x7f54d00039e0 2024-04-15T15:16:08.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.824+0000 7f54e27fc640 1 -- 172.21.15.70:0/250103437 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54d4005d80 con 0x7f54d00039e0 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 -- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d00039e0 msgr2=0x7f54d0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 --2- 172.21.15.70:0/250103437 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d00039e0 0x7f54d0003dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f54d4009ab0 tx=0x7f54d0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e37fe640 1 -- 172.21.15.70:0/250103437 reap_dead start 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 -- 172.21.15.70:0/250103437 shutdown_connections 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 -- 172.21.15.70:0/250103437 >> 172.21.15.70:0/250103437 conn(0x7f54e408bf60 msgr2=0x7f54e40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 -- 172.21.15.70:0/250103437 shutdown_connections 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.825+0000 7f54e957c640 1 -- 172.21.15.70:0/250103437 wait complete. 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.826+0000 7f54e957c640 1 Processor -- start 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.826+0000 7f54e957c640 1 -- start start 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d00039e0 0x7f54e40b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54e40b1230 0x7f54e40b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54e40d1d40 con 0x7f54e40ccde0 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54e40b4450 con 0x7f54e40b1230 2024-04-15T15:16:08.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f54e40d1ec0 con 0x7f54d00039e0 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41800/0 (socket says 172.21.15.70:41800) 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 -- 172.21.15.70:0/3308020272 learned_addr learned my addr 172.21.15.70:0/3308020272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d00039e0 0x7f54e40b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e37fe640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54e40b1230 0x7f54e40b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 -- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d00039e0 msgr2=0x7f54e40b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d00039e0 0x7f54e40b34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 -- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54e40b1230 msgr2=0x7f54e40b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:08.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54e40b1230 0x7f54e40b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 -- 172.21.15.70:0/3308020272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e40b7320 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54dbfff640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f54cc007fd0 tx=0x7f54cc0076e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00f9c0 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54cc018070 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00fd80 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54e40b7600 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.827+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54e40b7bc0 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.829+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54e40689e0 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.829+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54cc01d460 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.829+0000 7f54db7fe640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 0x7f54c406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.829+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f54cc08be90 con 0x7f54e40ccde0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.829+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 0x7f54c406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:08.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.830+0000 7f54e2ffd640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 0x7f54c406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54d4035000 tx=0x7f54d4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:08.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:08.834+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54cc059f50 con 0x7f54e40ccde0 2024-04-15T15:16:09.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.008+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54e40ba500 con 0x7f54e40ccde0 2024-04-15T15:16:09.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.009+0000 7f54db7fe640 1 -- 172.21.15.70:0/3308020272 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7f54cc059850 con 0x7f54e40ccde0 2024-04-15T15:16:09.011 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:09.011 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 msgr2=0x7f54c406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 0x7f54c406edb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f54d4035000 tx=0x7f54d4030000 comp rx=0 tx=0).stop 2024-04-15T15:16:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 msgr2=0x7f54e40cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f54cc007fd0 tx=0x7f54cc0076e0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 shutdown_connections 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54e40b1230 0x7f54e40b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54c406c8f0 0x7f54c406edb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54e40ccde0 0x7f54e40cd260 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 --2- 172.21.15.70:0/3308020272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d00039e0 0x7f54e40b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 >> 172.21.15.70:0/3308020272 conn(0x7f54e408bf60 msgr2=0x7f54e40d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 shutdown_connections 2024-04-15T15:16:09.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.011+0000 7f54e957c640 1 -- 172.21.15.70:0/3308020272 wait complete. 2024-04-15T15:16:09.025 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:09.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:09.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.129+0000 7f1f39d9d640 1 Processor -- start 2024-04-15T15:16:09.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 -- start start 2024-04-15T15:16:09.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1f3415e700 0x7f1f340b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f340d2360 0x7f1f340d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415eae0 0x7f1f340d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f1f340b53b0 con 0x7f1f340d2360 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1f340b47a0 con 0x7f1f3415e700 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f39d9d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1f340b4a80 con 0x7f1f3415eae0 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f2affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f340d2360 0x7f1f340d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.130+0000 7f1f2affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f1f340d2360 0x7f1f340d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41802/0 (socket says 172.21.15.70:41802) 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 -- 172.21.15.70:0/1255917273 learned_addr learned my addr 172.21.15.70:0/1255917273 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f33fff640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415eae0 0x7f1f340d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f337fe640 1 --2- 172.21.15.70:0/1255917273 >> v2:172.21.15.96:3300/0 conn(0x7f1f3415e700 0x7f1f340b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 -- 172.21.15.70:0/1255917273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415eae0 msgr2=0x7f1f340d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415eae0 0x7f1f340d30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 -- 172.21.15.70:0/1255917273 >> v2:172.21.15.96:3300/0 conn(0x7f1f3415e700 msgr2=0x7f1f340b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 --2- 172.21.15.70:0/1255917273 >> v2:172.21.15.96:3300/0 conn(0x7f1f3415e700 0x7f1f340b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.131+0000 7f1f2affd640 1 -- 172.21.15.70:0/1255917273 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f340b4d60 con 0x7f1f340d2360 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f2affd640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f340d2b60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406c6c0 tx=0x7f1f1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25a00edbc7e1137 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f32ffd640 1 -- 172.21.15.70:0/1255917273 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c035020 con 0x7f1f340d2360 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f32ffd640 1 -- 172.21.15.70:0/1255917273 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f1c031870 con 0x7f1f340d2360 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f32ffd640 1 -- 172.21.15.70:0/1255917273 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c031c40 con 0x7f1f340d2360 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 -- 172.21.15.70:0/1255917273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 msgr2=0x7f1f340d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f340d2b60 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406c6c0 tx=0x7f1f1c02b910 comp rx=0 tx=0).stop 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 -- 172.21.15.70:0/1255917273 shutdown_connections 2024-04-15T15:16:09.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415eae0 0x7f1f340d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/1255917273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f340d2b60 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/1255917273 >> v2:172.21.15.96:3300/0 conn(0x7f1f3415e700 0x7f1f340b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 -- 172.21.15.70:0/1255917273 >> 172.21.15.70:0/1255917273 conn(0x7f1f3408bf60 msgr2=0x7f1f340d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 -- 172.21.15.70:0/1255917273 shutdown_connections 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.132+0000 7f1f39d9d640 1 -- 172.21.15.70:0/1255917273 wait complete. 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.133+0000 7f1f39d9d640 1 Processor -- start 2024-04-15T15:16:09.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 -- start start 2024-04-15T15:16:09.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415e700 0x7f1f340b9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 0x7f1f340ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1f340b5d80 con 0x7f1f340d2360 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1f340b4450 con 0x7f1f3415eae0 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f39d9d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1f340d1d40 con 0x7f1f3415e700 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41810/0 (socket says 172.21.15.70:41810) 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.134+0000 7f1f337fe640 1 -- 172.21.15.70:0/438649016 learned_addr learned my addr 172.21.15.70:0/438649016 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f33fff640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 0x7f1f340ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f2affd640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415e700 0x7f1f340b9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 -- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415e700 msgr2=0x7f1f340b9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415e700 0x7f1f340b9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 -- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 msgr2=0x7f1f340ccb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 0x7f1f340ccb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 -- 172.21.15.70:0/438649016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f340cbae0 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f33fff640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 0x7f1f340ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f337fe640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400e8e0 tx=0x7f1f2400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400c9a0 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f2400cb40 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f340cbdc0 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24005990 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.135+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f340cc380 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.136+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f340689e0 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.136+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1f24005b30 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.137+0000 7f1f30ff9640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 0x7f1f1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.137+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1f2408c4b0 con 0x7f1f340d2360 2024-04-15T15:16:09.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.137+0000 7f1f2affd640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 0x7f1f1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.137+0000 7f1f2affd640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 0x7f1f1406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.141+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1f24017020 con 0x7f1f340d2360 2024-04-15T15:16:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.317+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1f340caee0 con 0x7f1f340d2360 2024-04-15T15:16:09.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.317+0000 7f1f30ff9640 1 -- 172.21.15.70:0/438649016 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f1f2405a4e0 con 0x7f1f340d2360 2024-04-15T15:16:09.319 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 msgr2=0x7f1f1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 0x7f1f1406edb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c036000 tx=0x7f1f1c03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 msgr2=0x7f1f34151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1f2400e8e0 tx=0x7f1f2400eda0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 shutdown_connections 2024-04-15T15:16:09.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1f3415eae0 0x7f1f340ccb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1f3415e700 0x7f1f340b9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1f1406c8f0 0x7f1f1406edb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 --2- 172.21.15.70:0/438649016 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1f340d2360 0x7f1f34151a80 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 >> 172.21.15.70:0/438649016 conn(0x7f1f3408bf60 msgr2=0x7f1f340d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 shutdown_connections 2024-04-15T15:16:09.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.319+0000 7f1f39d9d640 1 -- 172.21.15.70:0/438649016 wait complete. 2024-04-15T15:16:09.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:09.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.437+0000 7f6048f52640 1 Processor -- start 2024-04-15T15:16:09.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.437+0000 7f6048f52640 1 -- start start 2024-04-15T15:16:09.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6048f52640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60440d0690 0x7f60440d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6048f52640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f60440b3160 0x7f604415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6048f52640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60440d0fb0 con 0x7f604415ef70 2024-04-15T15:16:09.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6048f52640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f604415cb50 con 0x7f60440d0690 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6048f52640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f604415bfd0 con 0x7f60440b3160 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f604259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60440d0690 0x7f60440d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f604259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f60440d0690 0x7f60440d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41820/0 (socket says 172.21.15.70:41820) 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f604259c640 1 -- 172.21.15.70:0/4202796304 learned_addr learned my addr 172.21.15.70:0/4202796304 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.438+0000 7f6041d9b640 1 --2- 172.21.15.70:0/4202796304 >> v2:172.21.15.96:3300/0 conn(0x7f60440b3160 0x7f604415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604259c640 1 -- 172.21.15.70:0/4202796304 >> v2:172.21.15.96:3300/0 conn(0x7f60440b3160 msgr2=0x7f604415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604259c640 1 --2- 172.21.15.70:0/4202796304 >> v2:172.21.15.96:3300/0 conn(0x7f60440b3160 0x7f604415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604259c640 1 -- 172.21.15.70:0/4202796304 >> v1:172.21.15.96:6789/0 conn(0x7f604415ef70 legacy=0x7f604415f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604259c640 1 -- 172.21.15.70:0/4202796304 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604415c2b0 con 0x7f60440d0690 2024-04-15T15:16:09.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604259c640 1 --2- 172.21.15.70:0/4202796304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60440d0690 0x7f60440d0a70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604406fec0 tx=0x7f603402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a621d37354fdafd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.439+0000 7f604159a640 1 -- 172.21.15.70:0/4202796304 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603402cb20 con 0x7f60440d0690 2024-04-15T15:16:09.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f604159a640 1 -- 172.21.15.70:0/4202796304 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603402ccc0 con 0x7f60440d0690 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f604159a640 1 -- 172.21.15.70:0/4202796304 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6034034610 con 0x7f60440d0690 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 -- 172.21.15.70:0/4202796304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60440d0690 msgr2=0x7f60440d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 --2- 172.21.15.70:0/4202796304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60440d0690 0x7f60440d0a70 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f604406fec0 tx=0x7f603402bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 -- 172.21.15.70:0/4202796304 shutdown_connections 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 --2- 172.21.15.70:0/4202796304 >> v2:172.21.15.96:3300/0 conn(0x7f60440b3160 0x7f604415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 --2- 172.21.15.70:0/4202796304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f60440d0690 0x7f60440d0a70 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 -- 172.21.15.70:0/4202796304 >> 172.21.15.70:0/4202796304 conn(0x7f604406be00 msgr2=0x7f604406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 -- 172.21.15.70:0/4202796304 shutdown_connections 2024-04-15T15:16:09.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.440+0000 7f6048f52640 1 -- 172.21.15.70:0/4202796304 wait complete. 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.441+0000 7f6048f52640 1 Processor -- start 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 -- start start 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 0x7f60441529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60440d0690 0x7f6044148010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f60440b4fb0 con 0x7f604415ef70 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f604415c870 con 0x7f60440d0690 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6048f52640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f604415bbd0 con 0x7f60440b3160 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6042d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6042d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41836/0 (socket says 172.21.15.70:41836) 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.442+0000 7f6042d9d640 1 -- 172.21.15.70:0/294632287 learned_addr learned my addr 172.21.15.70:0/294632287 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f604259c640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 0x7f60441529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6041d9b640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60440d0690 0x7f6044148010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 -- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 msgr2=0x7f60441529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 0x7f60441529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 -- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60440d0690 msgr2=0x7f6044148010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60440d0690 0x7f6044148010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 -- 172.21.15.70:0/294632287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6044148770 con 0x7f604415ef70 2024-04-15T15:16:09.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f604259c640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 0x7f60441529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6042d9d640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c400 tx=0x7f603c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c017070 con 0x7f604415ef70 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c0052d0 con 0x7f604415ef70 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c003810 con 0x7f604415ef70 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6044149180 con 0x7f604415ef70 2024-04-15T15:16:09.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.443+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6044149740 con 0x7f604415ef70 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.444+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60440689e0 con 0x7f604415ef70 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.444+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f603c01a070 con 0x7f604415ef70 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.445+0000 7f603affd640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 0x7f602406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.445+0000 7f604259c640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 0x7f602406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.445+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f603c08b250 con 0x7f604415ef70 2024-04-15T15:16:09.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.445+0000 7f604259c640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 0x7f602406ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6034035000 tx=0x7f6034030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.451 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.449+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f603c059310 con 0x7f604415ef70 2024-04-15T15:16:09.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.624+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f6044149910 con 0x7f604415ef70 2024-04-15T15:16:09.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.625+0000 7f603affd640 1 -- 172.21.15.70:0/294632287 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f603c058c10 con 0x7f604415ef70 2024-04-15T15:16:09.627 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 msgr2=0x7f602406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 0x7f602406ede0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6034035000 tx=0x7f6034030000 comp rx=0 tx=0).stop 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 msgr2=0x7f6044168650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f603c00c400 tx=0x7f603c00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 shutdown_connections 2024-04-15T15:16:09.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f602406c920 0x7f602406ede0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f604415ef70 0x7f6044168650 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f60440d0690 0x7f6044148010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 --2- 172.21.15.70:0/294632287 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f60440b3160 0x7f60441529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 >> 172.21.15.70:0/294632287 conn(0x7f604406be00 msgr2=0x7f604415f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 shutdown_connections 2024-04-15T15:16:09.630 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.627+0000 7f6048f52640 1 -- 172.21.15.70:0/294632287 wait complete. 2024-04-15T15:16:09.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.745+0000 7fb572baa640 1 Processor -- start 2024-04-15T15:16:09.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.745+0000 7fb572baa640 1 -- start start 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb572baa640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb56c158770 0x7fb56c15bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb572baa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0b5230 0x7fb56c072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb572baa640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb56c073080 con 0x7fb56c072c60 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb572baa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb56c0b4da0 con 0x7fb56c0b5230 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb572baa640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb56c15c1d0 con 0x7fb56c158770 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb56bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0b5230 0x7fb56c072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb56bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb56c0b5230 0x7fb56c072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41846/0 (socket says 172.21.15.70:41846) 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.746+0000 7fb56bfff640 1 -- 172.21.15.70:0/1200179775 learned_addr learned my addr 172.21.15.70:0/1200179775 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb570946640 1 --2- 172.21.15.70:0/1200179775 >> v2:172.21.15.96:3300/0 conn(0x7fb56c158770 0x7fb56c15bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb56bfff640 1 -- 172.21.15.70:0/1200179775 >> v2:172.21.15.96:3300/0 conn(0x7fb56c158770 msgr2=0x7fb56c15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb56bfff640 1 --2- 172.21.15.70:0/1200179775 >> v2:172.21.15.96:3300/0 conn(0x7fb56c158770 0x7fb56c15bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb56bfff640 1 -- 172.21.15.70:0/1200179775 >> v1:172.21.15.96:6789/0 conn(0x7fb56c072c60 legacy=0x7fb56c15e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb56bfff640 1 -- 172.21.15.70:0/1200179775 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c0b4190 con 0x7fb56c0b5230 2024-04-15T15:16:09.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.747+0000 7fb56bfff640 1 --2- 172.21.15.70:0/1200179775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0b5230 0x7fb56c072720 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb56c06fe80 tx=0x7fb56002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39791971955c10 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1200179775 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb56002cb20 con 0x7fb56c0b5230 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1200179775 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb56002ccc0 con 0x7fb56c0b5230 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb56b7fe640 1 -- 172.21.15.70:0/1200179775 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb560034610 con 0x7fb56c0b5230 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 -- 172.21.15.70:0/1200179775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0b5230 msgr2=0x7fb56c072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 --2- 172.21.15.70:0/1200179775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0b5230 0x7fb56c072720 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb56c06fe80 tx=0x7fb56002bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 -- 172.21.15.70:0/1200179775 shutdown_connections 2024-04-15T15:16:09.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 --2- 172.21.15.70:0/1200179775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c0b5230 0x7fb56c072720 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 --2- 172.21.15.70:0/1200179775 >> v2:172.21.15.96:3300/0 conn(0x7fb56c158770 0x7fb56c15bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 -- 172.21.15.70:0/1200179775 >> 172.21.15.70:0/1200179775 conn(0x7fb56c06be00 msgr2=0x7fb56c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 -- 172.21.15.70:0/1200179775 shutdown_connections 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.748+0000 7fb572baa640 1 -- 172.21.15.70:0/1200179775 wait complete. 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.749+0000 7fb572baa640 1 Processor -- start 2024-04-15T15:16:09.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.749+0000 7fb572baa640 1 -- start start 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb56c072c60 0x7fb56c1508a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb56c0b5230 0x7fb56c0adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c0b4a30 con 0x7fb56c158770 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb56c0b4750 con 0x7fb56c0b5230 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb572baa640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb56c071fa0 con 0x7fb56c072c60 2024-04-15T15:16:09.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb571147640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb571147640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41858/0 (socket says 172.21.15.70:41858) 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.750+0000 7fb571147640 1 -- 172.21.15.70:0/53223804 learned_addr learned my addr 172.21.15.70:0/53223804 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb570946640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb56c072c60 0x7fb56c1508a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb56bfff640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb56c0b5230 0x7fb56c0adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 -- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb56c072c60 msgr2=0x7fb56c1508a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb56c072c60 0x7fb56c1508a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 -- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb56c0b5230 msgr2=0x7fb56c0adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb56c0b5230 0x7fb56c0adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 -- 172.21.15.70:0/53223804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb56c1e7fe0 con 0x7fb56c158770 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb571147640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb564001eb0 tx=0x7fb56400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564017070 con 0x7fb56c158770 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5640052d0 con 0x7fb56c158770 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb564003810 con 0x7fb56c158770 2024-04-15T15:16:09.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb56c0af030 con 0x7fb56c158770 2024-04-15T15:16:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.751+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb56c0af5f0 con 0x7fb56c158770 2024-04-15T15:16:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.752+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb56c0aedc0 con 0x7fb56c158770 2024-04-15T15:16:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.753+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb56401a070 con 0x7fb56c158770 2024-04-15T15:16:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.753+0000 7fb5697fa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 0x7fb55806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:09.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.753+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb56408b170 con 0x7fb56c158770 2024-04-15T15:16:09.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.753+0000 7fb570946640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 0x7fb55806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:09.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.753+0000 7fb570946640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 0x7fb55806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0079a0 tx=0x7fb55c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:09.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.757+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb56c1e7fe0 con 0x7fb56c158770 2024-04-15T15:16:09.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.933+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb56c0689e0 con 0x7fb56c158770 2024-04-15T15:16:09.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.933+0000 7fb5697fa640 1 -- 172.21.15.70:0/53223804 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb564058be0 con 0x7fb56c158770 2024-04-15T15:16:09.935 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:09.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 msgr2=0x7fb55806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 0x7fb55806edb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb55c0079a0 tx=0x7fb55c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:09.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 msgr2=0x7fb56c1e86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:09.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb564001eb0 tx=0x7fb56400c890 comp rx=0 tx=0).stop 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 shutdown_connections 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb55806c8f0 0x7fb55806edb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb56c158770 0x7fb56c1e86a0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb56c0b5230 0x7fb56c0adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 --2- 172.21.15.70:0/53223804 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb56c072c60 0x7fb56c1508a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 >> 172.21.15.70:0/53223804 conn(0x7fb56c06be00 msgr2=0x7fb56c0b2170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 shutdown_connections 2024-04-15T15:16:09.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:09.935+0000 7fb572baa640 1 -- 172.21.15.70:0/53223804 wait complete. 2024-04-15T15:16:09.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:10.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.052+0000 7f449b104640 1 Processor -- start 2024-04-15T15:16:10.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.052+0000 7f449b104640 1 -- start start 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f449b104640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f44940701f0 0x7f4494160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f449b104640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44940d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f449b104640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4494070720 con 0x7f449409a9d0 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f449b104640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f449409ba90 con 0x7f44940701f0 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f449b104640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f44940d08c0 con 0x7f449415fb30 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f4498ea0640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f449409a9d0 0x7f449409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51392/0 (socket says 172.21.15.70:51392) 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.053+0000 7f4498ea0640 1 -- 172.21.15.70:0/1358024510 learned_addr learned my addr 172.21.15.70:0/1358024510 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3891786695 0 0) 0x7f4494070720 con 0x7f449409a9d0 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44996a1640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44940d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4480003520 con 0x7f449409a9d0 2024-04-15T15:16:10.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4493fff640 1 --2- 172.21.15.70:0/1358024510 >> v2:172.21.15.96:3300/0 conn(0x7f44940701f0 0x7f4494160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2600344876 0 0) 0x7f4480003520 con 0x7f449409a9d0 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 msgr2=0x7f44940d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44940d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 >> v2:172.21.15.96:3300/0 conn(0x7f44940701f0 msgr2=0x7f4494160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 --2- 172.21.15.70:0/1358024510 >> v2:172.21.15.96:3300/0 conn(0x7f44940701f0 0x7f4494160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4494070a90 con 0x7f449409a9d0 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f44840019f0 con 0x7f449409a9d0 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44996a1640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44940d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 >> v1:172.21.15.70:6789/0 conn(0x7f449409a9d0 legacy=0x7f449409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44800039b0 0x7f4480003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4480004800 0x7f4480004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f44940d08c0 con 0x7f44800039b0 2024-04-15T15:16:10.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f448000b9f0 con 0x7f4480004800 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f448000bbf0 con 0x7f449415fb30 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4493fff640 1 --2- 172.21.15.70:0/1358024510 >> v2:172.21.15.96:3300/0 conn(0x7f44940701f0 0x7f4494160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44800039b0 0x7f4480003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f44996a1640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4493fff640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4480004800 0x7f4480004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 -- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 msgr2=0x7f44800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f449415fb30 0x7f44800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 -- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4480004800 msgr2=0x7f4480004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4480004800 0x7f4480004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.054+0000 7f4498ea0640 1 -- 172.21.15.70:0/1358024510 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4480003520 con 0x7f44800039b0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.055+0000 7f4498ea0640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44800039b0 0x7f4480003dd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f448400a8a0 tx=0x7f4480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad00561d2e52f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.055+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448400a4c0 con 0x7f44800039b0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.055+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4484034650 con 0x7f44800039b0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.055+0000 7f44937fe640 1 -- 172.21.15.70:0/1358024510 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4484034a40 con 0x7f44800039b0 2024-04-15T15:16:10.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 -- 172.21.15.70:0/1358024510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44800039b0 msgr2=0x7f4480003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 --2- 172.21.15.70:0/1358024510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44800039b0 0x7f4480003dd0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f448400a8a0 tx=0x7f4480005830 comp rx=0 tx=0).stop 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f44996a1640 1 -- 172.21.15.70:0/1358024510 reap_dead start 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 -- 172.21.15.70:0/1358024510 shutdown_connections 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 -- 172.21.15.70:0/1358024510 >> 172.21.15.70:0/1358024510 conn(0x7f44940d2460 msgr2=0x7f44940d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 -- 172.21.15.70:0/1358024510 shutdown_connections 2024-04-15T15:16:10.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.056+0000 7f449b104640 1 -- 172.21.15.70:0/1358024510 wait complete. 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.057+0000 7f449b104640 1 Processor -- start 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.057+0000 7f449b104640 1 -- start start 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 0x7f44941f5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 0x7f4494162860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f449415ffc0 con 0x7f44941f7880 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4494071050 con 0x7f44800039b0 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f449b104640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f44940b1c60 con 0x7f44941623e0 2024-04-15T15:16:10.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41870/0 (socket says 172.21.15.70:41870) 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 -- 172.21.15.70:0/296492054 learned_addr learned my addr 172.21.15.70:0/296492054 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f44996a1640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 0x7f4494162860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4498ea0640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 0x7f44941f5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 -- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 msgr2=0x7f4494162860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 0x7f4494162860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 -- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 msgr2=0x7f44941f5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 0x7f44941f5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 -- 172.21.15.70:0/296492054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44940c8150 con 0x7f44941f7880 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f44996a1640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 0x7f4494162860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4498ea0640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 0x7f44941f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.058+0000 7f4493fff640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f447c0108c0 tx=0x7f447c010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.059+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c01a070 con 0x7f44941f7880 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.059+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f447c00cb40 con 0x7f44941f7880 2024-04-15T15:16:10.065 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.059+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c00f040 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.059+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44940af380 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.059+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44940c8430 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.060+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4494162da0 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.060+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f447c01d070 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.060+0000 7f44917fa640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.061+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f447c08c2f0 con 0x7f44941f7880 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.061+0000 7f4498ea0640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.061+0000 7f4498ea0640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 0x7f447406ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4484034000 tx=0x7f4484009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.066 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.065+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f447c00ee60 con 0x7f44941f7880 2024-04-15T15:16:10.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.240+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f44940689e0 con 0x7f44941f7880 2024-04-15T15:16:10.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.240+0000 7f44917fa640 1 -- 172.21.15.70:0/296492054 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f447c05a3b0 con 0x7f44941f7880 2024-04-15T15:16:10.242 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:10.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 msgr2=0x7f447406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 0x7f447406ede0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4484034000 tx=0x7f4484009030 comp rx=0 tx=0).stop 2024-04-15T15:16:10.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 msgr2=0x7f44940c7900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f447c0108c0 tx=0x7f447c010d80 comp rx=0 tx=0).stop 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 shutdown_connections 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f44941623e0 0x7f4494162860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f447406c920 0x7f447406ede0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f44941f7880 0x7f44940c7900 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 --2- 172.21.15.70:0/296492054 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f44800039b0 0x7f44941f5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 >> 172.21.15.70:0/296492054 conn(0x7f44940d2460 msgr2=0x7f4494074be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 shutdown_connections 2024-04-15T15:16:10.245 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.242+0000 7f449b104640 1 -- 172.21.15.70:0/296492054 wait complete. 2024-04-15T15:16:10.256 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:10.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.359+0000 7f073379d640 1 Processor -- start 2024-04-15T15:16:10.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.359+0000 7f073379d640 1 -- start start 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f073379d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f072c0b3160 0x7f072c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f073379d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f073379d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f072c15cb50 con 0x7f072c0d0690 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f073379d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f072c15bfd0 con 0x7f072c0b3160 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f073379d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f072c15c2b0 con 0x7f072c0ae280 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f0730d38640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f072c0b3160 0x7f072c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f0730d38640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f072c0b3160 0x7f072c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41878/0 (socket says 172.21.15.70:41878) 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f0730d38640 1 -- 172.21.15.70:0/2308627018 learned_addr learned my addr 172.21.15.70:0/2308627018 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.360+0000 7f0731d3a640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.361+0000 7f0730d38640 1 -- 172.21.15.70:0/2308627018 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 msgr2=0x7f072c0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.361+0000 7f0730d38640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c0ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.361+0000 7f0730d38640 1 -- 172.21.15.70:0/2308627018 >> v1:172.21.15.96:6789/0 conn(0x7f072c0d0690 legacy=0x7f072c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.364 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.361+0000 7f0730d38640 1 -- 172.21.15.70:0/2308627018 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c15c590 con 0x7f072c0b3160 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.361+0000 7f0730d38640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0b3160 0x7f072c15b690 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f071c00bb10 tx=0x7f071c02e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbbe2f633779422 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f0723fff640 1 -- 172.21.15.70:0/2308627018 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c03b070 con 0x7f072c0b3160 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f0723fff640 1 -- 172.21.15.70:0/2308627018 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f071c036440 con 0x7f072c0b3160 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f0723fff640 1 -- 172.21.15.70:0/2308627018 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f071c036810 con 0x7f072c0b3160 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 -- 172.21.15.70:0/2308627018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0b3160 msgr2=0x7f072c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0b3160 0x7f072c15b690 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f071c00bb10 tx=0x7f071c02e150 comp rx=0 tx=0).stop 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 -- 172.21.15.70:0/2308627018 shutdown_connections 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 --2- 172.21.15.70:0/2308627018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0b3160 0x7f072c15b690 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 -- 172.21.15.70:0/2308627018 >> 172.21.15.70:0/2308627018 conn(0x7f072c06be00 msgr2=0x7f072c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 -- 172.21.15.70:0/2308627018 shutdown_connections 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.362+0000 7f073379d640 1 -- 172.21.15.70:0/2308627018 wait complete. 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.363+0000 7f073379d640 1 Processor -- start 2024-04-15T15:16:10.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.363+0000 7f073379d640 1 -- start start 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 0x7f072c0cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f072c15bbd0 con 0x7f072c0d0690 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f072c0b4fb0 con 0x7f072c0b3160 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f073379d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f072c15bd50 con 0x7f072c0ae280 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f0731d3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f0731d3a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41890/0 (socket says 172.21.15.70:41890) 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f0731d3a640 1 -- 172.21.15.70:0/1019649984 learned_addr learned my addr 172.21.15.70:0/1019649984 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f0731539640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.364+0000 7f0730d38640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 0x7f072c0cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 -- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 msgr2=0x7f072c156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 -- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 msgr2=0x7f072c0cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 0x7f072c0cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 -- 172.21.15.70:0/1019649984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f072c0ce840 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731539640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0730d38640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 0x7f072c0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0731d3a640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f072400d5d0 tx=0x7f072400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240125f0 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0724019070 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f072c0cee00 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07240129f0 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.365+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f072c0cf3c0 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.366+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f072c0689e0 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.366+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f0724012b90 con 0x7f072c0d0690 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.367+0000 7f0721ffb640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 0x7f071006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.367+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f072408c330 con 0x7f072c0d0690 2024-04-15T15:16:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.367+0000 7f0731539640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 0x7f071006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.367+0000 7f0731539640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 0x7f071006edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f072c0ccb00 tx=0x7f0728009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.371+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f072405a370 con 0x7f072c0d0690 2024-04-15T15:16:10.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.563+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f072c0739e0 con 0x7f072c0d0690 2024-04-15T15:16:10.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.563+0000 7f0721ffb640 1 -- 172.21.15.70:0/1019649984 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f0724060030 con 0x7f072c0d0690 2024-04-15T15:16:10.565 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:10.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.565+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 msgr2=0x7f071006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.565+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 0x7f071006edb0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f072c0ccb00 tx=0x7f0728009040 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.565+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 msgr2=0x7f072c1544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.565+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f072400d5d0 tx=0x7f072400da90 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 shutdown_connections 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f071006c8f0 0x7f071006edb0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f072c0d0690 0x7f072c1544e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f072c0b3160 0x7f072c0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 --2- 172.21.15.70:0/1019649984 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f072c0ae280 0x7f072c156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 >> 172.21.15.70:0/1019649984 conn(0x7f072c06be00 msgr2=0x7f072c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 shutdown_connections 2024-04-15T15:16:10.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.566+0000 7f073379d640 1 -- 172.21.15.70:0/1019649984 wait complete. 2024-04-15T15:16:10.578 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-15T15:16:10.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:10.579 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:10.615 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:16:10.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:10.615 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:16:10.673 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-15T15:16:10.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:10.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.774+0000 7f1072d04640 1 Processor -- start 2024-04-15T15:16:10.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 -- start start 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f106c15eb00 0x7f106c0ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f106c0b0560 0x7f106c06b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f106c0cbd10 con 0x7f106c0b09a0 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f106c15eee0 con 0x7f106c15eb00 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.775+0000 7f1072d04640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f106c0cbff0 con 0x7f106c0b0560 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f106c15eb00 0x7f106c0ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f106c15eb00 0x7f106c0ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41892/0 (socket says 172.21.15.70:41892) 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 -- 172.21.15.70:0/3909711964 learned_addr learned my addr 172.21.15.70:0/3909711964 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:10.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 -- 172.21.15.70:0/3909711964 >> v2:172.21.15.96:3300/0 conn(0x7f106c0b0560 msgr2=0x7f106c06b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 --2- 172.21.15.70:0/3909711964 >> v2:172.21.15.96:3300/0 conn(0x7f106c0b0560 0x7f106c06b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 -- 172.21.15.70:0/3909711964 >> v1:172.21.15.96:6789/0 conn(0x7f106c0b09a0 legacy=0x7f106c0ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:10.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.776+0000 7f1070aa0640 1 -- 172.21.15.70:0/3909711964 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c0cc2d0 con 0x7f106c15eb00 2024-04-15T15:16:10.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1070aa0640 1 --2- 172.21.15.70:0/3909711964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c15eb00 0x7f106c0ae2a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f106c0cf0b0 tx=0x7f105c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ab1a13f9cf285b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f106b7fe640 1 -- 172.21.15.70:0/3909711964 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c035020 con 0x7f106c15eb00 2024-04-15T15:16:10.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f106b7fe640 1 -- 172.21.15.70:0/3909711964 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f105c031870 con 0x7f106c15eb00 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f106b7fe640 1 -- 172.21.15.70:0/3909711964 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c031c40 con 0x7f106c15eb00 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 -- 172.21.15.70:0/3909711964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c15eb00 msgr2=0x7f106c0ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 --2- 172.21.15.70:0/3909711964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c15eb00 0x7f106c0ae2a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f106c0cf0b0 tx=0x7f105c02b910 comp rx=0 tx=0).stop 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 -- 172.21.15.70:0/3909711964 shutdown_connections 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 --2- 172.21.15.70:0/3909711964 >> v2:172.21.15.96:3300/0 conn(0x7f106c0b0560 0x7f106c06b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 --2- 172.21.15.70:0/3909711964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c15eb00 0x7f106c0ae2a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 -- 172.21.15.70:0/3909711964 >> 172.21.15.70:0/3909711964 conn(0x7f106c09a9d0 msgr2=0x7f106c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 -- 172.21.15.70:0/3909711964 shutdown_connections 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.777+0000 7f1072d04640 1 -- 172.21.15.70:0/3909711964 wait complete. 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.778+0000 7f1072d04640 1 Processor -- start 2024-04-15T15:16:10.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 -- start start 2024-04-15T15:16:10.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106c0b0560 0x7f106c154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f106c15eb00 0x7f106c0c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0ae7e0 con 0x7f106c0b09a0 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f106c0cc890 con 0x7f106c0b0560 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.779+0000 7f1072d04640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f106c0cec60 con 0x7f106c15eb00 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41898/0 (socket says 172.21.15.70:41898) 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 -- 172.21.15.70:0/2535603285 learned_addr learned my addr 172.21.15.70:0/2535603285 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f1070aa0640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106c0b0560 0x7f106c154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f10712a1640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f106c15eb00 0x7f106c0c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 -- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f106c15eb00 msgr2=0x7f106c0c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f106c15eb00 0x7f106c0c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 -- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106c0b0560 msgr2=0x7f106c154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106c0b0560 0x7f106c154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 -- 172.21.15.70:0/2535603285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f106c154fa0 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f106bfff640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f106000b4a0 tx=0x7f106000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060014070 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1060010610 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f106c0c8700 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1060010a00 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.780+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f106c0c8cc0 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.781+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f106c0690f0 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.782+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1060017070 con 0x7f106c0b09a0 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.782+0000 7f10697fa640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 0x7f105806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:10.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.782+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f106008b930 con 0x7f106c0b09a0 2024-04-15T15:16:10.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.782+0000 7f1070aa0640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 0x7f105806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:10.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.783+0000 7f1070aa0640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 0x7f105806edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f105c036000 tx=0x7f105c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:10.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:10.787+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f106c154fa0 con 0x7f106c0b09a0 2024-04-15T15:16:11.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.000+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f106c06f800 con 0x7f106c0b09a0 2024-04-15T15:16:11.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.000+0000 7f10697fa640 1 -- 172.21.15.70:0/2535603285 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f1060059960 con 0x7f106c0b09a0 2024-04-15T15:16:11.002 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:11.003 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:16:11.003 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:16:11.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 msgr2=0x7f105806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 0x7f105806edb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f105c036000 tx=0x7f105c03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 msgr2=0x7f106c0ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f106000b4a0 tx=0x7f106000b960 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 shutdown_connections 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f106c15eb00 0x7f106c0c94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f105806c8f0 0x7f105806edb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f106c0b09a0 0x7f106c0ca2a0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 --2- 172.21.15.70:0/2535603285 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f106c0b0560 0x7f106c154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 >> 172.21.15.70:0/2535603285 conn(0x7f106c09a9d0 msgr2=0x7f106c0af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 shutdown_connections 2024-04-15T15:16:11.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.002+0000 7f1072d04640 1 -- 172.21.15.70:0/2535603285 wait complete. 2024-04-15T15:16:11.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:11.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.116+0000 7fbc7f9f2640 1 Processor -- start 2024-04-15T15:16:11.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7f9f2640 1 -- start start 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7f9f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbc780d1a10 0x7fbc780b6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7f9f2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc7815ba70 con 0x7fbc7815b690 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7f9f2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc780ce5b0 con 0x7fbc780d08a0 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7f9f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbc780ce890 con 0x7fbc780d1a10 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7cf8d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbc780d1a10 0x7fbc780b6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7d78e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbc7815b690 0x7fbc780d1300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51396/0 (socket says 172.21.15.70:51396) 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.117+0000 7fbc7d78e640 1 -- 172.21.15.70:0/3691097846 learned_addr learned my addr 172.21.15.70:0/3691097846 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.118+0000 7fbc7cf8d640 1 -- 172.21.15.70:0/3691097846 >> v1:172.21.15.96:6789/0 conn(0x7fbc780d08a0 legacy=0x7fbc780b1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.118+0000 7fbc7cf8d640 1 -- 172.21.15.70:0/3691097846 >> v1:172.21.15.70:6789/0 conn(0x7fbc7815b690 legacy=0x7fbc780d1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.118+0000 7fbc7cf8d640 1 -- 172.21.15.70:0/3691097846 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc780ceb70 con 0x7fbc780d1a10 2024-04-15T15:16:11.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.118+0000 7fbc77fff640 1 -- 172.21.15.70:0/3691097846 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1380221336 0 0) 0x7fbc68001490 con 0x7fbc7815b690 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7cf8d640 1 --2- 172.21.15.70:0/3691097846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d1a10 0x7fbc780b6ff0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0097d0 tx=0x7fbc6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80e4c22651b69a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc77fff640 1 -- 172.21.15.70:0/3691097846 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c035020 con 0x7fbc780d1a10 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc77fff640 1 -- 172.21.15.70:0/3691097846 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6c031830 con 0x7fbc780d1a10 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc77fff640 1 -- 172.21.15.70:0/3691097846 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6c031c00 con 0x7fbc780d1a10 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/3691097846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d1a10 msgr2=0x7fbc780b6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/3691097846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d1a10 0x7fbc780b6ff0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c0097d0 tx=0x7fbc6c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/3691097846 shutdown_connections 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/3691097846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d1a10 0x7fbc780b6ff0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/3691097846 >> 172.21.15.70:0/3691097846 conn(0x7fbc7809a9d0 msgr2=0x7fbc780b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/3691097846 shutdown_connections 2024-04-15T15:16:11.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.119+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/3691097846 wait complete. 2024-04-15T15:16:11.124 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.120+0000 7fbc7f9f2640 1 Processor -- start 2024-04-15T15:16:11.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 -- start start 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbc780d1a10 0x7fbc7806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 0x7fbc7806f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbc780b1d70 con 0x7fbc780d08a0 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbc780d1010 con 0x7fbc7815b690 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7f9f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbc780b1ef0 con 0x7fbc780d1a10 2024-04-15T15:16:11.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7d78e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7d78e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41928/0 (socket says 172.21.15.70:41928) 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.121+0000 7fbc7d78e640 1 -- 172.21.15.70:0/1340394699 learned_addr learned my addr 172.21.15.70:0/1340394699 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7cf8d640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbc780d1a10 0x7fbc7806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7df8f640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 0x7fbc7806f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 -- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbc780d1a10 msgr2=0x7fbc7806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbc780d1a10 0x7fbc7806efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 -- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 msgr2=0x7fbc7806f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 0x7fbc7806f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 -- 172.21.15.70:0/1340394699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc78145070 con 0x7fbc780d08a0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7df8f640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 0x7fbc7806f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7d78e640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800c7c0 tx=0x7fbc6800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc68018070 con 0x7fbc780d08a0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc6800a640 con 0x7fbc780d08a0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc781442b0 con 0x7fbc780d08a0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc6800aa30 con 0x7fbc780d08a0 2024-04-15T15:16:11.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.122+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc78144870 con 0x7fbc780d08a0 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.123+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc780b1b50 con 0x7fbc780d08a0 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.124+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbc6801b070 con 0x7fbc780d08a0 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.124+0000 7fbc75ffb640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 0x7fbc6406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.124+0000 7fbc7cf8d640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 0x7fbc6406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.124+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbc6808b460 con 0x7fbc780d08a0 2024-04-15T15:16:11.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.125+0000 7fbc7cf8d640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 0x7fbc6406ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c036000 tx=0x7fbc6c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.129+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbc780b1b50 con 0x7fbc780d08a0 2024-04-15T15:16:11.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.303+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc780690f0 con 0x7fbc780d08a0 2024-04-15T15:16:11.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.304+0000 7fbc75ffb640 1 -- 172.21.15.70:0/1340394699 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fbc7815ba70 con 0x7fbc780d08a0 2024-04-15T15:16:11.306 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:11.307 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:11.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 msgr2=0x7fbc6406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 0x7fbc6406ede0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbc6c036000 tx=0x7fbc6c03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 msgr2=0x7fbc78141940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbc6800c7c0 tx=0x7fbc6800cc80 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 shutdown_connections 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbc7815b690 0x7fbc7806f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbc780d1a10 0x7fbc7806efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbc6406c920 0x7fbc6406ede0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 --2- 172.21.15.70:0/1340394699 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbc780d08a0 0x7fbc78141940 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 >> 172.21.15.70:0/1340394699 conn(0x7fbc7809a9d0 msgr2=0x7fbc78156190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 shutdown_connections 2024-04-15T15:16:11.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.306+0000 7fbc7f9f2640 1 -- 172.21.15.70:0/1340394699 wait complete. 2024-04-15T15:16:11.320 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:15:08.083098+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:15:08.589040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-15T15:15:12.300503+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-15T15:16:11.320 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:11.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.424+0000 7f4050d32640 1 Processor -- start 2024-04-15T15:16:11.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.424+0000 7f4050d32640 1 -- start start 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4050d32640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f404c0af340 0x7f404c0b4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4050d32640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f404c0b1620 0x7f404c0b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4050d32640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f404c0af920 con 0x7f404c15ecb0 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4050d32640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f404c15f090 con 0x7f404c0b1620 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4050d32640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f404c0cdf80 con 0x7f404c0af340 2024-04-15T15:16:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f404c0b1620 0x7f404c0b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f404c15ecb0 0x7f404c0b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51404/0 (socket says 172.21.15.70:51404) 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404a59c640 1 -- 172.21.15.70:0/1333602434 learned_addr learned my addr 172.21.15.70:0/1333602434 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4041d9b640 1 --2- 172.21.15.70:0/1333602434 >> v2:172.21.15.96:3300/0 conn(0x7f404c0af340 0x7f404c0b4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4049d9b640 1 -- 172.21.15.70:0/1333602434 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3838980849 0 0) 0x7f404c0af920 con 0x7f404c15ecb0 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f4049d9b640 1 -- 172.21.15.70:0/1333602434 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4038003520 con 0x7f404c15ecb0 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404ad9d640 1 -- 172.21.15.70:0/1333602434 >> v2:172.21.15.96:3300/0 conn(0x7f404c0af340 msgr2=0x7f404c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404ad9d640 1 --2- 172.21.15.70:0/1333602434 >> v2:172.21.15.96:3300/0 conn(0x7f404c0af340 0x7f404c0b4350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404ad9d640 1 -- 172.21.15.70:0/1333602434 >> v1:172.21.15.70:6789/0 conn(0x7f404c15ecb0 legacy=0x7f404c0b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.425+0000 7f404ad9d640 1 -- 172.21.15.70:0/1333602434 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404c0afc00 con 0x7f404c0b1620 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f404ad9d640 1 --2- 172.21.15.70:0/1333602434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0b1620 0x7f404c0b1ab0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f404c06a050 tx=0x7f404402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d806c5afeb0a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f4049d9b640 1 -- 172.21.15.70:0/1333602434 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044039070 con 0x7f404c0b1620 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f4049d9b640 1 -- 172.21.15.70:0/1333602434 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f404402cd00 con 0x7f404c0b1620 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f4049d9b640 1 -- 172.21.15.70:0/1333602434 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4044034660 con 0x7f404c0b1620 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f4050d32640 1 -- 172.21.15.70:0/1333602434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0b1620 msgr2=0x7f404c0b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.426+0000 7f4050d32640 1 --2- 172.21.15.70:0/1333602434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0b1620 0x7f404c0b1ab0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f404c06a050 tx=0x7f404402bf50 comp rx=0 tx=0).stop 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 -- 172.21.15.70:0/1333602434 shutdown_connections 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 --2- 172.21.15.70:0/1333602434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0b1620 0x7f404c0b1ab0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 --2- 172.21.15.70:0/1333602434 >> v2:172.21.15.96:3300/0 conn(0x7f404c0af340 0x7f404c0b4350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 -- 172.21.15.70:0/1333602434 >> 172.21.15.70:0/1333602434 conn(0x7f404c0b6870 msgr2=0x7f404c0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 -- 172.21.15.70:0/1333602434 shutdown_connections 2024-04-15T15:16:11.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.427+0000 7f4050d32640 1 -- 172.21.15.70:0/1333602434 wait complete. 2024-04-15T15:16:11.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.428+0000 7f4050d32640 1 Processor -- start 2024-04-15T15:16:11.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.428+0000 7f4050d32640 1 -- start start 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 0x7f404c153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 0x7f404c06faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f404c0ce410 con 0x7f404c0af340 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f404c0b4890 con 0x7f404c15ecb0 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4050d32640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f404c0ce590 con 0x7f404c0b1620 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41944/0 (socket says 172.21.15.70:41944) 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 -- 172.21.15.70:0/2195163444 learned_addr learned my addr 172.21.15.70:0/2195163444 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4041d9b640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 0x7f404c153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 -- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 msgr2=0x7f404c153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404ad9d640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 0x7f404c06faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 0x7f404c153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 -- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 msgr2=0x7f404c06faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 0x7f404c06faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 -- 172.21.15.70:0/2195163444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f404c1f8940 con 0x7f404c0af340 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f4041d9b640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 0x7f404c153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404ad9d640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 0x7f404c06faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f404a59c640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c3c0 tx=0x7f403c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c017070 con 0x7f404c0af340 2024-04-15T15:16:11.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.429+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403c005ce0 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.430+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f404c1f8c20 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.430+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f403c0072f0 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.430+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f404c0ca770 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.430+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f404c0689e0 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.431+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f403c01a070 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.431+0000 7f40437fe640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 0x7f402c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.432+0000 7f4041d9b640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 0x7f402c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.432+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f403c08b2b0 con 0x7f404c0af340 2024-04-15T15:16:11.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.432+0000 7f4041d9b640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 0x7f402c06edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f404c15cc00 tx=0x7f4034008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.436+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f403c059370 con 0x7f404c0af340 2024-04-15T15:16:11.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.613+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f404c070c80 con 0x7f404c0af340 2024-04-15T15:16:11.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.614+0000 7f40437fe640 1 -- 172.21.15.70:0/2195163444 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f403c05f020 con 0x7f404c0af340 2024-04-15T15:16:11.616 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:11.618 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 msgr2=0x7f402c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 0x7f402c06edb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f404c15cc00 tx=0x7f4034008040 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 msgr2=0x7f404c1f80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f403c00c3c0 tx=0x7f403c00c880 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 shutdown_connections 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f404c15ecb0 0x7f404c06faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f404c0b1620 0x7f404c153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f402c06c8f0 0x7f402c06edb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 --2- 172.21.15.70:0/2195163444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f404c0af340 0x7f404c1f80d0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 >> 172.21.15.70:0/2195163444 conn(0x7f404c0b6870 msgr2=0x7f404c0b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 shutdown_connections 2024-04-15T15:16:11.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.616+0000 7f4050d32640 1 -- 172.21.15.70:0/2195163444 wait complete. 2024-04-15T15:16:11.629 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:11.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.773+0000 7fc6eda1b640 1 Processor -- start 2024-04-15T15:16:11.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.773+0000 7fc6eda1b640 1 -- start start 2024-04-15T15:16:11.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6eda1b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6e80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6eda1b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e80b47a0 con 0x7fc6e80d2360 2024-04-15T15:16:11.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6eda1b640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6e80b4a80 con 0x7fc6e80d3270 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6eda1b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80b53b0 con 0x7fc6e815e700 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6de7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc6e80d2360 0x7fc6e80d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51406/0 (socket says 172.21.15.70:51406) 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6de7fc640 1 -- 172.21.15.70:0/1361925153 learned_addr learned my addr 172.21.15.70:0/1361925153 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734263704 0 0) 0x7fc6e80b47a0 con 0x7fc6e80d2360 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6d4003520 con 0x7fc6e80d2360 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.774+0000 7fc6e6ffd640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6e80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3843751522 0 0) 0x7fc6d4003520 con 0x7fc6e80d2360 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 msgr2=0x7fc6e80b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6e80b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 >> v1:172.21.15.96:6789/0 conn(0x7fc6e80d3270 legacy=0x7fc6e80cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e80b4d60 con 0x7fc6e80d2360 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fc6d0001670 con 0x7fc6e80d2360 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 >> v1:172.21.15.70:6789/0 conn(0x7fc6e80d2360 legacy=0x7fc6e80d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6d40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6d4004010 0x7fc6d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4004d40 0x7fc6d40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80b53b0 con 0x7fc6d4004d40 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d400bc10 con 0x7fc6d4004010 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d400be10 con 0x7fc6e815e700 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4004d40 0x7fc6d40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6de7fc640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6d4004010 0x7fc6d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e6ffd640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6d40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 -- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 msgr2=0x7fc6d40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6e815e700 0x7fc6d40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 -- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6d4004010 msgr2=0x7fc6d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6d4004010 0x7fc6d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.775+0000 7fc6e77fe640 1 -- 172.21.15.70:0/1361925153 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d4003520 con 0x7fc6d4004d40 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4004d40 0x7fc6d40051c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001e50 tx=0x7fc6d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad518729d1be4f1d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0001a70 con 0x7fc6d4004d40 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e0003b70 con 0x7fc6d4004d40 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6e67fc640 1 -- 172.21.15.70:0/1361925153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00050d0 con 0x7fc6d4004d40 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6eda1b640 1 -- 172.21.15.70:0/1361925153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4004d40 msgr2=0x7fc6d40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/1361925153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4004d40 0x7fc6d40051c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001e50 tx=0x7fc6d4005830 comp rx=0 tx=0).stop 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.776+0000 7fc6e77fe640 1 -- 172.21.15.70:0/1361925153 reap_dead start 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.777+0000 7fc6eda1b640 1 -- 172.21.15.70:0/1361925153 shutdown_connections 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.777+0000 7fc6eda1b640 1 -- 172.21.15.70:0/1361925153 >> 172.21.15.70:0/1361925153 conn(0x7fc6e808bf60 msgr2=0x7fc6e80d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.777+0000 7fc6eda1b640 1 -- 172.21.15.70:0/1361925153 shutdown_connections 2024-04-15T15:16:11.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.777+0000 7fc6eda1b640 1 -- 172.21.15.70:0/1361925153 wait complete. 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.778+0000 7fc6eda1b640 1 Processor -- start 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.778+0000 7fc6eda1b640 1 -- start start 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6d4004d40 0x7fc6e815c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6e81f8870 0x7fc6e81f7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80d1d40 con 0x7fc6e81f81f0 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc6e80b4450 con 0x7fc6e81f8870 2024-04-15T15:16:11.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc6e80d1ec0 con 0x7fc6d4004d40 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41956/0 (socket says 172.21.15.70:41956) 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 -- 172.21.15.70:0/558470988 learned_addr learned my addr 172.21.15.70:0/558470988 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e6ffd640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6d4004d40 0x7fc6e815c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6de7fc640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6e81f8870 0x7fc6e81f7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 -- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6d4004d40 msgr2=0x7fc6e815c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6d4004d40 0x7fc6e815c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 -- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6e81f8870 msgr2=0x7fc6e81f7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6e81f8870 0x7fc6e81f7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 -- 172.21.15.70:0/558470988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6e80b9cd0 con 0x7fc6e81f81f0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e77fe640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001f70 tx=0x7fc6e00014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0007db0 con 0x7fc6e81f81f0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e0003970 con 0x7fc6e81f81f0 2024-04-15T15:16:11.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6e80cc100 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0006d70 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.779+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6e80cc6c0 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.780+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6e80689e0 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.781+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc6e0041030 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.781+0000 7fc6e4ff9640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.781+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc6e00acf20 con 0x7fc6e81f81f0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.781+0000 7fc6e6ffd640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:11.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.782+0000 7fc6e6ffd640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 0x7fc6c806edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005f40 tx=0x7fc6d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:11.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.786+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc6e0033020 con 0x7fc6e81f81f0 2024-04-15T15:16:11.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.961+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc6e80ba500 con 0x7fc6e81f81f0 2024-04-15T15:16:11.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.961+0000 7fc6e4ff9640 1 -- 172.21.15.70:0/558470988 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc6e007fe20 con 0x7fc6e81f81f0 2024-04-15T15:16:11.963 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 msgr2=0x7fc6c806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 0x7fc6c806edb0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8005f40 tx=0x7fc6d8008040 comp rx=0 tx=0).stop 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 msgr2=0x7fc6e80b7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0001f70 tx=0x7fc6e00014a0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 shutdown_connections 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc6c806c8f0 0x7fc6c806edb0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6e81f81f0 0x7fc6e80b7130 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc6e81f8870 0x7fc6e81f7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 --2- 172.21.15.70:0/558470988 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6d4004d40 0x7fc6e815c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:11.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 >> 172.21.15.70:0/558470988 conn(0x7fc6e808bf60 msgr2=0x7fc6e806d3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:11.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 shutdown_connections 2024-04-15T15:16:11.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:11.963+0000 7fc6eda1b640 1 -- 172.21.15.70:0/558470988 wait complete. 2024-04-15T15:16:11.976 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.079+0000 7f4836d1c640 1 Processor -- start 2024-04-15T15:16:12.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.079+0000 7f4836d1c640 1 -- start start 2024-04-15T15:16:12.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4836d1c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f48300b3160 0x7f483015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4836d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f48300ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4836d1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f483015cb50 con 0x7f48300d0690 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4836d1c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f483015bfd0 con 0x7f48300b3160 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4836d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f483015c2b0 con 0x7f48300ae280 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4834ab8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f48300d0690 0x7f48300d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:51422/0 (socket says 172.21.15.70:51422) 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f4834ab8640 1 -- 172.21.15.70:0/2863834655 learned_addr learned my addr 172.21.15.70:0/2863834655 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f48352b9640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f48300ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482ffff640 1 --2- 172.21.15.70:0/2863834655 >> v2:172.21.15.96:3300/0 conn(0x7f48300b3160 0x7f483015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482f7fe640 1 -- 172.21.15.70:0/2863834655 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1601640280 0 0) 0x7f483015cb50 con 0x7f48300d0690 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482f7fe640 1 -- 172.21.15.70:0/2863834655 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f481c003520 con 0x7f48300d0690 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482ffff640 1 -- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 msgr2=0x7f48300ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482ffff640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f48300ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.080+0000 7f482ffff640 1 -- 172.21.15.70:0/2863834655 >> v1:172.21.15.70:6789/0 conn(0x7f48300d0690 legacy=0x7f48300d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.081+0000 7f482ffff640 1 -- 172.21.15.70:0/2863834655 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483015c590 con 0x7f48300b3160 2024-04-15T15:16:12.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.081+0000 7f48352b9640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f48300ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.081+0000 7f482ffff640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300b3160 0x7f483015b690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4820009dc0 tx=0x7f482002be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aea6e75e64a6fd28 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f482f7fe640 1 -- 172.21.15.70:0/2863834655 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820039070 con 0x7f48300b3160 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f482f7fe640 1 -- 172.21.15.70:0/2863834655 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482002cc40 con 0x7f48300b3160 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f482f7fe640 1 -- 172.21.15.70:0/2863834655 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820034440 con 0x7f48300b3160 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 -- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300b3160 msgr2=0x7f483015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300b3160 0x7f483015b690 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4820009dc0 tx=0x7f482002be90 comp rx=0 tx=0).stop 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 -- 172.21.15.70:0/2863834655 shutdown_connections 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f48300ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 --2- 172.21.15.70:0/2863834655 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300b3160 0x7f483015b690 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 -- 172.21.15.70:0/2863834655 >> 172.21.15.70:0/2863834655 conn(0x7f483006be00 msgr2=0x7f483006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 -- 172.21.15.70:0/2863834655 shutdown_connections 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.082+0000 7f4836d1c640 1 -- 172.21.15.70:0/2863834655 wait complete. 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.083+0000 7f4836d1c640 1 Processor -- start 2024-04-15T15:16:12.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.083+0000 7f4836d1c640 1 -- start start 2024-04-15T15:16:12.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f4830149580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 0x7f4830159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f483015bbd0 con 0x7f48300b3160 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f48300b4fb0 con 0x7f48300d0690 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f4836d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f483015bd50 con 0x7f48300ae280 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f482ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f482ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41968/0 (socket says 172.21.15.70:41968) 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.084+0000 7f482ffff640 1 -- 172.21.15.70:0/4065011064 learned_addr learned my addr 172.21.15.70:0/4065011064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f4834ab8640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f4830149580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f48352b9640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 0x7f4830159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 -- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 msgr2=0x7f4830149580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f4830149580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 -- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 msgr2=0x7f4830159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 0x7f4830159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 -- 172.21.15.70:0/4065011064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4830138f00 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f4834ab8640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f4830149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f48352b9640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 0x7f4830159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482ffff640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4820037040 tx=0x7f4820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820039050 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4820045070 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4820009450 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48301391e0 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.085+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48301397a0 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.086+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4830073ca0 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.087+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f48200406b0 con 0x7f48300b3160 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.087+0000 7f482d7fa640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 0x7f481006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.087+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f48200ae640 con 0x7f48300b3160 2024-04-15T15:16:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.087+0000 7f4834ab8640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 0x7f481006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.088+0000 7f4834ab8640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 0x7f481006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4828005880 tx=0x7f4828009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.092+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4830073ca0 con 0x7f48300b3160 2024-04-15T15:16:12.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.265+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48300689e0 con 0x7f48300b3160 2024-04-15T15:16:12.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.266+0000 7f482d7fa640 1 -- 172.21.15.70:0/4065011064 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f482004f020 con 0x7f48300b3160 2024-04-15T15:16:12.268 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 msgr2=0x7f481006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 0x7f481006edb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4828005880 tx=0x7f4828009040 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 msgr2=0x7f4830150210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f4820037040 tx=0x7f4820004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 shutdown_connections 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f48300d0690 0x7f4830159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f481006c8f0 0x7f481006edb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f48300b3160 0x7f4830150210 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 --2- 172.21.15.70:0/4065011064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f48300ae280 0x7f4830149580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 >> 172.21.15.70:0/4065011064 conn(0x7f483006be00 msgr2=0x7f483006e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 shutdown_connections 2024-04-15T15:16:12.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.268+0000 7f4836d1c640 1 -- 172.21.15.70:0/4065011064 wait complete. 2024-04-15T15:16:12.280 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-15T15:16:12.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.385+0000 7fe6f0a0d640 1 Processor -- start 2024-04-15T15:16:12.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.385+0000 7fe6f0a0d640 1 -- start start 2024-04-15T15:16:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6f0a0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6f0a0d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6ec0b1d30 0x7fe6ec0af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6f0a0d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6ec0b10d0 con 0x7fe6ec0b4350 2024-04-15T15:16:12.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6f0a0d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe6ec15f090 con 0x7fe6ec0b1d30 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6f0a0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec0cdf80 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ead9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe6ec0b1d30 0x7fe6ec0af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6e9d9b640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe6ec0b4350 0x7fe6ec0b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:45152/0 (socket says 172.21.15.70:45152) 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:42394/0 (socket says 172.21.15.70:42394) 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6e9d9b640 1 -- 172.21.15.70:0/2835956569 learned_addr learned my addr 172.21.15.70:0/2835956569 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6e959a640 1 -- 172.21.15.70:0/2835956569 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439884148 0 0) 0x7fe6ec0b10d0 con 0x7fe6ec0b4350 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 -- 172.21.15.70:0/2835956569 >> v2:172.21.15.96:3300/0 conn(0x7fe6ec0b1d30 msgr2=0x7fe6ec0af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 --2- 172.21.15.70:0/2835956569 >> v2:172.21.15.96:3300/0 conn(0x7fe6ec0b1d30 0x7fe6ec0af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 -- 172.21.15.70:0/2835956569 >> v1:172.21.15.96:6789/0 conn(0x7fe6ec0b4350 legacy=0x7fe6ec0b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.386+0000 7fe6ea59c640 1 -- 172.21.15.70:0/2835956569 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec0af920 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.387+0000 7fe6ea59c640 1 --2- 172.21.15.70:0/2835956569 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097a0 tx=0x7fe6dc02b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5beb9d31238d29b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.387+0000 7fe6e959a640 1 -- 172.21.15.70:0/2835956569 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc034070 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.387+0000 7fe6e959a640 1 -- 172.21.15.70:0/2835956569 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6dc030610 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.387+0000 7fe6e959a640 1 -- 172.21.15.70:0/2835956569 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc0309e0 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/2835956569 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 msgr2=0x7fe6ec0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/2835956569 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc0097a0 tx=0x7fe6dc02b960 comp rx=0 tx=0).stop 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/2835956569 shutdown_connections 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/2835956569 >> v2:172.21.15.96:3300/0 conn(0x7fe6ec0b1d30 0x7fe6ec0af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/2835956569 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec15ecb0 0x7fe6ec0b0b90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/2835956569 >> 172.21.15.70:0/2835956569 conn(0x7fe6ec0b6870 msgr2=0x7fe6ec0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/2835956569 shutdown_connections 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.388+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/2835956569 wait complete. 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.389+0000 7fe6f0a0d640 1 Processor -- start 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.389+0000 7fe6f0a0d640 1 -- start start 2024-04-15T15:16:12.395 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ec0b1d30 0x7fe6ec15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec0b4350 0x7fe6ec158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec0b4790 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe6ec0afee0 con 0x7fe6ec0b1d30 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6f0a0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe6ec0b4910 con 0x7fe6ec0b4350 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6ead9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6ead9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:41970/0 (socket says 172.21.15.70:41970) 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6ead9d640 1 -- 172.21.15.70:0/3781995152 learned_addr learned my addr 172.21.15.70:0/3781995152 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6ea59c640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ec0b1d30 0x7fe6ec15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.390+0000 7fe6e9d9b640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec0b4350 0x7fe6ec158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 -- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec0b4350 msgr2=0x7fe6ec158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec0b4350 0x7fe6ec158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 -- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ec0b1d30 msgr2=0x7fe6ec15f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.396 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ec0b1d30 0x7fe6ec15f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 -- 172.21.15.70:0/3781995152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6ec0cb7b0 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6ead9d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400efc0 tx=0x7fe6e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4018020 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6e4009b10 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6ec15f860 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6e40146c0 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.391+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6ec15fe20 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.392+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6ec0689e0 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.393+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe6e4005d10 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.393+0000 7fe6daffd640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 0x7fe6cc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.393+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe6e408cb20 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.397 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.393+0000 7fe6ea59c640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 0x7fe6cc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.398 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.394+0000 7fe6ea59c640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 0x7fe6cc06edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc036000 tx=0x7fe6dc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.399 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.398+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe6e4022030 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.574+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe6ec0b6df0 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.576 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.575+0000 7fe6daffd640 1 -- 172.21.15.70:0/3781995152 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fe6e4010040 con 0x7fe6ec15ecb0 2024-04-15T15:16:12.577 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:12.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 msgr2=0x7fe6cc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 0x7fe6cc06edb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc036000 tx=0x7fe6dc008030 comp rx=0 tx=0).stop 2024-04-15T15:16:12.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 msgr2=0x7fe6ec0cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400efc0 tx=0x7fe6e400c490 comp rx=0 tx=0).stop 2024-04-15T15:16:12.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 shutdown_connections 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe6cc06c8f0 0x7fe6cc06edb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe6ec15ecb0 0x7fe6ec0cb270 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe6ec0b4350 0x7fe6ec158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 --2- 172.21.15.70:0/3781995152 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe6ec0b1d30 0x7fe6ec15f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 >> 172.21.15.70:0/3781995152 conn(0x7fe6ec0b6870 msgr2=0x7fe6ec0d18c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 shutdown_connections 2024-04-15T15:16:12.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.577+0000 7fe6f0a0d640 1 -- 172.21.15.70:0/3781995152 wait complete. 2024-04-15T15:16:12.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.692+0000 7f0506152640 1 Processor -- start 2024-04-15T15:16:12.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.692+0000 7f0506152640 1 -- start start 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 0x7f0500071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f05000af3e0 0x7f05000af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05000d0690 0x7f05000d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f050015db40 con 0x7f05000d0690 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f050015cfc0 con 0x7f05000af3e0 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f0506152640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f050015d2a0 con 0x7f0500072d80 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05000d0690 0x7f05000d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f05000d0690 0x7f05000d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44798/0 (socket says 172.21.15.70:44798) 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 -- 172.21.15.70:0/417423986 learned_addr learned my addr 172.21.15.70:0/417423986 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04feffd640 1 --2- 172.21.15.70:0/417423986 >> v2:172.21.15.96:3300/0 conn(0x7f05000af3e0 0x7f05000af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04ff7fe640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 0x7f0500071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 -- 172.21.15.70:0/417423986 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 msgr2=0x7f0500071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 0x7f0500071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 -- 172.21.15.70:0/417423986 >> v2:172.21.15.96:3300/0 conn(0x7f05000af3e0 msgr2=0x7f05000af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 --2- 172.21.15.70:0/417423986 >> v2:172.21.15.96:3300/0 conn(0x7f05000af3e0 0x7f05000af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04fffff640 1 -- 172.21.15.70:0/417423986 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f050015d580 con 0x7f05000d0690 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.693+0000 7f04ff7fe640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 0x7f0500071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.694+0000 7f04fffff640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05000d0690 0x7f05000d0b20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f05000d1270 tx=0x7f04f802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7015bb5d87a7bc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.694+0000 7f04fe7fc640 1 -- 172.21.15.70:0/417423986 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f802c9e0 con 0x7f05000d0690 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.694+0000 7f04fe7fc640 1 -- 172.21.15.70:0/417423986 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f802cb80 con 0x7f05000d0690 2024-04-15T15:16:12.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.694+0000 7f04fe7fc640 1 -- 172.21.15.70:0/417423986 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f80343f0 con 0x7f05000d0690 2024-04-15T15:16:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 -- 172.21.15.70:0/417423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05000d0690 msgr2=0x7f05000d0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05000d0690 0x7f05000d0b20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f05000d1270 tx=0x7f04f802be40 comp rx=0 tx=0).stop 2024-04-15T15:16:12.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 -- 172.21.15.70:0/417423986 shutdown_connections 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05000d0690 0x7f05000d0b20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 --2- 172.21.15.70:0/417423986 >> v2:172.21.15.96:3300/0 conn(0x7f05000af3e0 0x7f05000af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 --2- 172.21.15.70:0/417423986 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0500072d80 0x7f0500071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 -- 172.21.15.70:0/417423986 >> 172.21.15.70:0/417423986 conn(0x7f050006be00 msgr2=0x7f050006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 -- 172.21.15.70:0/417423986 shutdown_connections 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.695+0000 7f0506152640 1 -- 172.21.15.70:0/417423986 wait complete. 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.696+0000 7f0506152640 1 Processor -- start 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.696+0000 7f0506152640 1 -- start start 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05000af3e0 0x7f0500159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f05000d0690 0x7f05001599c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f050015cd10 con 0x7f0500072d80 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f050015cb90 con 0x7f05000af3e0 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f050006e060 con 0x7f05000d0690 2024-04-15T15:16:12.701 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44806/0 (socket says 172.21.15.70:44806) 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 -- 172.21.15.70:0/3983561837 learned_addr learned my addr 172.21.15.70:0/3983561837 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 -- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f05000d0690 msgr2=0x7f05001599c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f05000d0690 0x7f05001599c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 -- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05000af3e0 msgr2=0x7f0500159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04feffd640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05000af3e0 0x7f0500159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05000af3e0 0x7f0500159480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 -- 172.21.15.70:0/3983561837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05000cd890 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04ff7fe640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f04f000e810 tx=0x7f04f000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f000f040 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04f0004030 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05000cdb70 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.697+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04f0004400 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.698+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05001571c0 con 0x7f0500072d80 2024-04-15T15:16:12.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.698+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05000ce090 con 0x7f0500072d80 2024-04-15T15:16:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.699+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04f000c6d0 con 0x7f0500072d80 2024-04-15T15:16:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.699+0000 7f04e7fff640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 0x7f04ec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.699+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f04f008bc50 con 0x7f0500072d80 2024-04-15T15:16:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.699+0000 7f04feffd640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 0x7f04ec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:12.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.700+0000 7f04feffd640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 0x7f04ec06edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f05000ce490 tx=0x7f04f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:12.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.704+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f05000ce090 con 0x7f0500072d80 2024-04-15T15:16:12.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.914+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05000689e0 con 0x7f0500072d80 2024-04-15T15:16:12.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.914+0000 7f04e7fff640 1 -- 172.21.15.70:0/3983561837 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f04f0091020 con 0x7f0500072d80 2024-04-15T15:16:12.916 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:12.916 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:16:12.916 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 msgr2=0x7f04ec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 0x7f04ec06edb0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f05000ce490 tx=0x7f04f4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 msgr2=0x7f0500147810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f04f000e810 tx=0x7f04f000ecd0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 shutdown_connections 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f05000d0690 0x7f05001599c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f05000af3e0 0x7f0500159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04ec06c8f0 0x7f04ec06edb0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 --2- 172.21.15.70:0/3983561837 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0500072d80 0x7f0500147810 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 >> 172.21.15.70:0/3983561837 conn(0x7f050006be00 msgr2=0x7f0500156780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 shutdown_connections 2024-04-15T15:16:12.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:12.916+0000 7f0506152640 1 -- 172.21.15.70:0/3983561837 wait complete. 2024-04-15T15:16:12.929 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:13.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.031+0000 7f54af6f2640 1 Processor -- start 2024-04-15T15:16:13.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.031+0000 7f54af6f2640 1 -- start start 2024-04-15T15:16:13.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54af6f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54a8071d10 0x7f54a80d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54af6f2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a815db40 con 0x7f54a8072d80 2024-04-15T15:16:13.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54af6f2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54a8072170 con 0x7f54a80d0bd0 2024-04-15T15:16:13.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54af6f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f54a80d1040 con 0x7f54a8071d10 2024-04-15T15:16:13.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54ad48e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f54a8072d80 0x7f54a80af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54568/0 (socket says 172.21.15.70:54568) 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54acc8d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54a8071d10 0x7f54a80d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54ad48e640 1 -- 172.21.15.70:0/4133974616 learned_addr learned my addr 172.21.15.70:0/4133974616 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54acc8d640 1 -- 172.21.15.70:0/4133974616 >> v1:172.21.15.96:6789/0 conn(0x7f54a80d0bd0 legacy=0x7f54a806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54acc8d640 1 -- 172.21.15.70:0/4133974616 >> v1:172.21.15.70:6789/0 conn(0x7f54a8072d80 legacy=0x7f54a80af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f549ffff640 1 -- 172.21.15.70:0/4133974616 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1276181374 0 0) 0x7f54a815db40 con 0x7f54a8072d80 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.032+0000 7f54acc8d640 1 -- 172.21.15.70:0/4133974616 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a815cfc0 con 0x7f54a8071d10 2024-04-15T15:16:13.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.033+0000 7f54acc8d640 1 --2- 172.21.15.70:0/4133974616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a8071d10 0x7f54a80d0690 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f549802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=271cba5e81ace63c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.033+0000 7f549ffff640 1 -- 172.21.15.70:0/4133974616 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54980315a0 con 0x7f54a8071d10 2024-04-15T15:16:13.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.033+0000 7f549ffff640 1 -- 172.21.15.70:0/4133974616 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5498037070 con 0x7f54a8071d10 2024-04-15T15:16:13.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.033+0000 7f549ffff640 1 -- 172.21.15.70:0/4133974616 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5498031960 con 0x7f54a8071d10 2024-04-15T15:16:13.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 -- 172.21.15.70:0/4133974616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a8071d10 msgr2=0x7f54a80d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 --2- 172.21.15.70:0/4133974616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a8071d10 0x7f54a80d0690 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f54980097a0 tx=0x7f549802b840 comp rx=0 tx=0).stop 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 -- 172.21.15.70:0/4133974616 shutdown_connections 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 --2- 172.21.15.70:0/4133974616 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a8071d10 0x7f54a80d0690 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 -- 172.21.15.70:0/4133974616 >> 172.21.15.70:0/4133974616 conn(0x7f54a806be00 msgr2=0x7f54a806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 -- 172.21.15.70:0/4133974616 shutdown_connections 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.034+0000 7f54af6f2640 1 -- 172.21.15.70:0/4133974616 wait complete. 2024-04-15T15:16:13.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.035+0000 7f54af6f2640 1 Processor -- start 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.035+0000 7f54af6f2640 1 -- start start 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 0x7f54a81530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54a8072d80 0x7f54a80caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54a815d860 con 0x7f54a80d0bd0 2024-04-15T15:16:13.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54a815d580 con 0x7f54a8072d80 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f54a80afaf0 con 0x7f54a8071d10 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44812/0 (socket says 172.21.15.70:44812) 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 -- 172.21.15.70:0/718739012 learned_addr learned my addr 172.21.15.70:0/718739012 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54ad48e640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 0x7f54a81530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54acc8d640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54a8072d80 0x7f54a80caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 -- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 msgr2=0x7f54a81530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 0x7f54a81530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 -- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54a8072d80 msgr2=0x7f54a80caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54a8072d80 0x7f54a80caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 -- 172.21.15.70:0/718739012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54a81f43c0 con 0x7f54a80d0bd0 2024-04-15T15:16:13.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54ad48e640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 0x7f54a81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54adc8f640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f54a000b410 tx=0x7f54a000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0015030 con 0x7f54a80d0bd0 2024-04-15T15:16:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54a0004020 con 0x7f54a80d0bd0 2024-04-15T15:16:13.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54a0004410 con 0x7f54a80d0bd0 2024-04-15T15:16:13.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.036+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54a81f4630 con 0x7f54a80d0bd0 2024-04-15T15:16:13.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.037+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54a81f4ed0 con 0x7f54a80d0bd0 2024-04-15T15:16:13.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.038+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54a81f4a20 con 0x7f54a80d0bd0 2024-04-15T15:16:13.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.038+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f54a0018070 con 0x7f54a80d0bd0 2024-04-15T15:16:13.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.038+0000 7f549dffb640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 0x7f548c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.039+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f54a008b9d0 con 0x7f54a80d0bd0 2024-04-15T15:16:13.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.039+0000 7f54ad48e640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 0x7f548c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.039+0000 7f54ad48e640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 0x7f548c06edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54a4007fd0 tx=0x7f54a4005b70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.043+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54a0059a00 con 0x7f54a80d0bd0 2024-04-15T15:16:13.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.218+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f54a80689e0 con 0x7f54a80d0bd0 2024-04-15T15:16:13.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.219+0000 7f549dffb640 1 -- 172.21.15.70:0/718739012 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7f54a0059300 con 0x7f54a80d0bd0 2024-04-15T15:16:13.221 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:13.221 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 msgr2=0x7f548c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 0x7f548c06edb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f54a4007fd0 tx=0x7f54a4005b70 comp rx=0 tx=0).stop 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 msgr2=0x7f54a80cb3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f54a000b410 tx=0x7f54a000b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 shutdown_connections 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f548c06c8f0 0x7f548c06edb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54a80d0bd0 0x7f54a80cb3f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54a8072d80 0x7f54a80caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 --2- 172.21.15.70:0/718739012 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54a8071d10 0x7f54a81530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 >> 172.21.15.70:0/718739012 conn(0x7f54a806be00 msgr2=0x7f54a80b2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 shutdown_connections 2024-04-15T15:16:13.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.221+0000 7f54af6f2640 1 -- 172.21.15.70:0/718739012 wait complete. 2024-04-15T15:16:13.234 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:16:13.235 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:13.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.338+0000 7f6e259c8640 1 Processor -- start 2024-04-15T15:16:13.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.338+0000 7f6e259c8640 1 -- start start 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e259c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 0x7f6e20071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e259c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2015db40 con 0x7f6e200af3e0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e259c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e2015cfc0 con 0x7f6e200d0da0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e259c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2015d2a0 con 0x7f6e20072d80 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e1e7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6e200af3e0 0x7f6e200d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54574/0 (socket says 172.21.15.70:54574) 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e1e7fc640 1 -- 172.21.15.70:0/3677305570 learned_addr learned my addr 172.21.15.70:0/3677305570 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e1effd640 1 --2- 172.21.15.70:0/3677305570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 0x7f6e20071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.339+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119995312 0 0) 0x7f6e2015db40 con 0x7f6e200af3e0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e08003520 con 0x7f6e200af3e0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121053086 0 0) 0x7f6e2015cfc0 con 0x7f6e200d0da0 2024-04-15T15:16:13.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6e2015db40 con 0x7f6e200d0da0 2024-04-15T15:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1effd640 1 -- 172.21.15.70:0/3677305570 >> v1:172.21.15.96:6789/0 conn(0x7f6e200d0da0 legacy=0x7f6e2006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1effd640 1 -- 172.21.15.70:0/3677305570 >> v1:172.21.15.70:6789/0 conn(0x7f6e200af3e0 legacy=0x7f6e200d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1292173333 0 0) 0x7f6e08003520 con 0x7f6e200af3e0 2024-04-15T15:16:13.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.340+0000 7f6e1effd640 1 -- 172.21.15.70:0/3677305570 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2015d580 con 0x7f6e20072d80 2024-04-15T15:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e1effd640 1 --2- 172.21.15.70:0/3677305570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 0x7f6e20071d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6e10009dc0 tx=0x7f6e1002bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd2526e92fe79a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e10039070 con 0x7f6e20072d80 2024-04-15T15:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1002c880 con 0x7f6e20072d80 2024-04-15T15:16:13.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e1dffb640 1 -- 172.21.15.70:0/3677305570 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14001670 con 0x7f6e20072d80 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 -- 172.21.15.70:0/3677305570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 msgr2=0x7f6e20071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 --2- 172.21.15.70:0/3677305570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 0x7f6e20071d10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6e10009dc0 tx=0x7f6e1002bd50 comp rx=0 tx=0).stop 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 -- 172.21.15.70:0/3677305570 shutdown_connections 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 --2- 172.21.15.70:0/3677305570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e20072d80 0x7f6e20071d10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 -- 172.21.15.70:0/3677305570 >> 172.21.15.70:0/3677305570 conn(0x7f6e2006be00 msgr2=0x7f6e2006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 -- 172.21.15.70:0/3677305570 shutdown_connections 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.341+0000 7f6e259c8640 1 -- 172.21.15.70:0/3677305570 wait complete. 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.342+0000 7f6e259c8640 1 Processor -- start 2024-04-15T15:16:13.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 -- start start 2024-04-15T15:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e20072d80 0x7f6e2014c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e200af3e0 0x7f6e20158cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2015cd10 con 0x7f6e200d0da0 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2015cb90 con 0x7f6e20072d80 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e259c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6e200af840 con 0x7f6e200af3e0 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e1f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e1f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44822/0 (socket says 172.21.15.70:44822) 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e1f7fe640 1 -- 172.21.15.70:0/1064740396 learned_addr learned my addr 172.21.15.70:0/1064740396 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.343+0000 7f6e1e7fc640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e200af3e0 0x7f6e20158cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1effd640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e20072d80 0x7f6e2014c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 -- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e200af3e0 msgr2=0x7f6e20158cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e200af3e0 0x7f6e20158cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 -- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e20072d80 msgr2=0x7f6e2014c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e20072d80 0x7f6e2014c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 -- 172.21.15.70:0/1064740396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e20159e70 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e1f7fe640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6e18001eb0 tx=0x7f6e1800c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e18017070 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e180052d0 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e180037f0 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2014cc10 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.344+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e200c9ba0 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.345+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e200689e0 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.346+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6e1801a070 con 0x7f6e200d0da0 2024-04-15T15:16:13.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.346+0000 7f6e07fff640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 0x7f6e0c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.346+0000 7f6e1effd640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 0x7f6e0c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.346+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6e1808b190 con 0x7f6e200d0da0 2024-04-15T15:16:13.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.347+0000 7f6e1effd640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 0x7f6e0c06edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e10036000 tx=0x7f6e10030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.351+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6e2014cc10 con 0x7f6e200d0da0 2024-04-15T15:16:13.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.563+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6e200b1ae0 con 0x7f6e200d0da0 2024-04-15T15:16:13.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.564+0000 7f6e07fff640 1 -- 172.21.15.70:0/1064740396 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7f6e20159e70 con 0x7f6e200d0da0 2024-04-15T15:16:13.566 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:13.566 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:16:13.566 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 msgr2=0x7f6e0c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 0x7f6e0c06edb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6e10036000 tx=0x7f6e10030000 comp rx=0 tx=0).stop 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 msgr2=0x7f6e200ad140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6e18001eb0 tx=0x7f6e1800c820 comp rx=0 tx=0).stop 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 shutdown_connections 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6e0c06c8f0 0x7f6e0c06edb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6e200d0da0 0x7f6e200ad140 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6e200af3e0 0x7f6e20158cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 --2- 172.21.15.70:0/1064740396 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6e20072d80 0x7f6e2014c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 >> 172.21.15.70:0/1064740396 conn(0x7f6e2006be00 msgr2=0x7f6e200b1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 shutdown_connections 2024-04-15T15:16:13.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.566+0000 7f6e259c8640 1 -- 172.21.15.70:0/1064740396 wait complete. 2024-04-15T15:16:13.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.680+0000 7ffa37293640 1 Processor -- start 2024-04-15T15:16:13.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.680+0000 7ffa37293640 1 -- start start 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa37293640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffa30158770 0x7ffa3015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa37293640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa3006dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa37293640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa3015c1d0 con 0x7ffa300b5230 2024-04-15T15:16:13.683 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa37293640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ffa300b4da0 con 0x7ffa30158770 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa37293640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ffa300b4190 con 0x7ffa30072e30 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffa30158770 0x7ffa3015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ffa30158770 0x7ffa3015bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44836/0 (socket says 172.21.15.70:44836) 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 -- 172.21.15.70:0/2152689371 learned_addr learned my addr 172.21.15.70:0/2152689371 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 -- 172.21.15.70:0/2152689371 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 msgr2=0x7ffa3006dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 --2- 172.21.15.70:0/2152689371 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa3006dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 -- 172.21.15.70:0/2152689371 >> v1:172.21.15.96:6789/0 conn(0x7ffa300b5230 legacy=0x7ffa30072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:13.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.681+0000 7ffa3502f640 1 -- 172.21.15.70:0/2152689371 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa300b4470 con 0x7ffa30158770 2024-04-15T15:16:13.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.682+0000 7ffa3502f640 1 --2- 172.21.15.70:0/2152689371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3015bc90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009870 tx=0x7ffa2c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f83cdc96e8d196 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.682+0000 7ffa27fff640 1 -- 172.21.15.70:0/2152689371 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c035020 con 0x7ffa30158770 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.682+0000 7ffa27fff640 1 -- 172.21.15.70:0/2152689371 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2c031640 con 0x7ffa30158770 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.682+0000 7ffa27fff640 1 -- 172.21.15.70:0/2152689371 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2c02d040 con 0x7ffa30158770 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 -- 172.21.15.70:0/2152689371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 msgr2=0x7ffa3015bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 --2- 172.21.15.70:0/2152689371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3015bc90 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c009870 tx=0x7ffa2c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 -- 172.21.15.70:0/2152689371 shutdown_connections 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 --2- 172.21.15.70:0/2152689371 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa3006dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 --2- 172.21.15.70:0/2152689371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3015bc90 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 -- 172.21.15.70:0/2152689371 >> 172.21.15.70:0/2152689371 conn(0x7ffa3006be00 msgr2=0x7ffa3006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 -- 172.21.15.70:0/2152689371 shutdown_connections 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.683+0000 7ffa37293640 1 -- 172.21.15.70:0/2152689371 wait complete. 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.684+0000 7ffa37293640 1 Processor -- start 2024-04-15T15:16:13.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.684+0000 7ffa37293640 1 -- start start 2024-04-15T15:16:13.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa301520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa300b5230 0x7ffa3009a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa30071fa0 con 0x7ffa30158770 2024-04-15T15:16:13.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ffa300b4a30 con 0x7ffa300b5230 2024-04-15T15:16:13.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa37293640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ffa30158b50 con 0x7ffa30072e30 2024-04-15T15:16:13.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44848/0 (socket says 172.21.15.70:44848) 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 -- 172.21.15.70:0/438570273 learned_addr learned my addr 172.21.15.70:0/438570273 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa3502f640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa301520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa3482e640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa300b5230 0x7ffa3009a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 -- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 msgr2=0x7ffa301520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa301520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 -- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa300b5230 msgr2=0x7ffa3009a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa300b5230 0x7ffa3009a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 -- 172.21.15.70:0/438570273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa3013d590 con 0x7ffa30158770 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.685+0000 7ffa35830640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800b9e0 tx=0x7ffa2800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.686+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28019070 con 0x7ffa30158770 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.686+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2800cb40 con 0x7ffa30158770 2024-04-15T15:16:13.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.686+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28014440 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.686+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa3013d870 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.686+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa3013de30 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.687+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa300689e0 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.687+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ffa2801c070 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.687+0000 7ffa25ffb640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 0x7ffa1406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.688+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffa2808bba0 con 0x7ffa30158770 2024-04-15T15:16:13.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.688+0000 7ffa3502f640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 0x7ffa1406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:13.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.688+0000 7ffa3502f640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 0x7ffa1406edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c036000 tx=0x7ffa2c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:13.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.692+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ffa28059c60 con 0x7ffa30158770 2024-04-15T15:16:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.905+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa30167a90 con 0x7ffa30158770 2024-04-15T15:16:13.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.905+0000 7ffa25ffb640 1 -- 172.21.15.70:0/438570273 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5174 (secure 0 0 0) 0x7ffa28059560 con 0x7ffa30158770 2024-04-15T15:16:13.907 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:13.908 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6833/1604355677","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1604355677},{"type":"v1","addr":"172.21.15.96:6833","nonce":1604355677}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4276,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/2155405036","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6834","nonce":2155405036},{"type":"v1","addr":"172.21.15.96:6835","nonce":2155405036}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4290,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/392693204","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":392693204},{"type":"v1","addr":"172.21.15.70:6836","nonce":392693204}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:15:09.639145+0000","modified":"2024-04-15T15:15:25.681548+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4293},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4293":{"gid":4293,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.70:6837/3048652815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":3048652815},{"type":"v1","addr":"172.21.15.70:6837","nonce":3048652815}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4293,"qdb_cluster":[4293]},"id":1}]} 2024-04-15T15:16:13.908 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 8 2024-04-15T15:16:13.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 msgr2=0x7ffa1406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 0x7ffa1406edb0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffa2c036000 tx=0x7ffa2c03c040 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 msgr2=0x7ffa3013ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa2800b9e0 tx=0x7ffa2800bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 shutdown_connections 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ffa1406c8f0 0x7ffa1406edb0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa30158770 0x7ffa3013ca20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ffa300b5230 0x7ffa3009a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.907+0000 7ffa37293640 1 --2- 172.21.15.70:0/438570273 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ffa30072e30 0x7ffa301520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.908+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 >> 172.21.15.70:0/438570273 conn(0x7ffa3006be00 msgr2=0x7ffa3015e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.908+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 shutdown_connections 2024-04-15T15:16:13.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:13.908+0000 7ffa37293640 1 -- 172.21.15.70:0/438570273 wait complete. 2024-04-15T15:16:13.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.022+0000 7fe346863640 1 Processor -- start 2024-04-15T15:16:14.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.022+0000 7fe346863640 1 -- start start 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe346863640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe34015e830 0x7fe3400b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe346863640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 0x7fe340067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe346863640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3400b3c10 con 0x7fe340067c60 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe346863640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe3400b3ef0 con 0x7fe34015e830 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe346863640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe3400b41d0 con 0x7fe34015ec10 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe33f7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 0x7fe340067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe33ffff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe34015e830 0x7fe3400b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe33ffff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe34015e830 0x7fe3400b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:33388/0 (socket says 172.21.15.70:33388) 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe33f7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 0x7fe340067720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:36122/0 (socket says 172.21.15.70:36122) 2024-04-15T15:16:14.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.023+0000 7fe33ffff640 1 -- 172.21.15.70:0/90226011 learned_addr learned my addr 172.21.15.70:0/90226011 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33ffff640 1 -- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 msgr2=0x7fe340067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33ffff640 1 --2- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 0x7fe340067720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33ffff640 1 -- 172.21.15.70:0/90226011 >> v1:172.21.15.96:6789/0 conn(0x7fe340067c60 legacy=0x7fe3400b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33ffff640 1 -- 172.21.15.70:0/90226011 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3400b44b0 con 0x7fe34015e830 2024-04-15T15:16:14.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33effd640 1 -- 172.21.15.70:0/90226011 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4217014094 0 0) 0x7fe3400b3c10 con 0x7fe340067c60 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.024+0000 7fe33ffff640 1 --2- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe34015e830 0x7fe3400b4cb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe330009870 tx=0x7fe33002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58f79268e77d6f35 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe33effd640 1 -- 172.21.15.70:0/90226011 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330034070 con 0x7fe34015e830 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe33effd640 1 -- 172.21.15.70:0/90226011 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe330030840 con 0x7fe34015e830 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe33effd640 1 -- 172.21.15.70:0/90226011 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330030c10 con 0x7fe34015e830 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 -- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe34015e830 msgr2=0x7fe3400b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 --2- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe34015e830 0x7fe3400b4cb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe330009870 tx=0x7fe33002b980 comp rx=0 tx=0).stop 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 -- 172.21.15.70:0/90226011 shutdown_connections 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 --2- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015ec10 0x7fe340067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 --2- 172.21.15.70:0/90226011 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe34015e830 0x7fe3400b4cb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 -- 172.21.15.70:0/90226011 >> 172.21.15.70:0/90226011 conn(0x7fe340073360 msgr2=0x7fe34006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 -- 172.21.15.70:0/90226011 shutdown_connections 2024-04-15T15:16:14.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.025+0000 7fe346863640 1 -- 172.21.15.70:0/90226011 wait complete. 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.026+0000 7fe346863640 1 Processor -- start 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 -- start start 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015e830 0x7fe3401f7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3401f83a0 0x7fe340141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe3401600e0 con 0x7fe340067c60 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe34015ff60 con 0x7fe3401f83a0 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.027+0000 7fe346863640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe3400b5ad0 con 0x7fe34015e830 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44862/0 (socket says 172.21.15.70:44862) 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 -- 172.21.15.70:0/765411052 learned_addr learned my addr 172.21.15.70:0/765411052 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe344e00640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3401f83a0 0x7fe340141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33f7fe640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015e830 0x7fe3401f7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 -- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015e830 msgr2=0x7fe3401f7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015e830 0x7fe3401f7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 -- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3401f83a0 msgr2=0x7fe340141e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3401f83a0 0x7fe340141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 -- 172.21.15.70:0/765411052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3401f8dc0 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33ffff640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe330009870 tx=0x7fe3300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330034040 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe330030640 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3401f8850 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe330039d40 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.028+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe340142cf0 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.029+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3400ae3c0 con 0x7fe340067c60 2024-04-15T15:16:14.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.030+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe33004b060 con 0x7fe340067c60 2024-04-15T15:16:14.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.030+0000 7fe33cff9640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 0x7fe32c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.030+0000 7fe33f7fe640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 0x7fe32c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.031+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe3300adc00 con 0x7fe340067c60 2024-04-15T15:16:14.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.031+0000 7fe33f7fe640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 0x7fe32c06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe34015de90 tx=0x7fe334008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.035+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe3400ae3c0 con 0x7fe340067c60 2024-04-15T15:16:14.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.211+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3400689e0 con 0x7fe340067c60 2024-04-15T15:16:14.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.212+0000 7fe33cff9640 1 -- 172.21.15.70:0/765411052 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18842 (secure 0 0 0) 0x7fe3300b4080 con 0x7fe340067c60 2024-04-15T15:16:14.214 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:14.214 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":27,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:15:51.403613+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:15:08.083098+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-15T15:15:08.589040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-15T15:15:12.300503+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":1}]},{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.23","mappings":[{"from":3,"to":2}]},{"pgid":"3.30","mappings":[{"from":5,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.1c","mappings":[{"from":1,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 msgr2=0x7fe32c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 0x7fe32c06edb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe34015de90 tx=0x7fe334008040 comp rx=0 tx=0).stop 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 msgr2=0x7fe3401e54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fe330009870 tx=0x7fe3300046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 shutdown_connections 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe3401f83a0 0x7fe340141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe34015e830 0x7fe3401f7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe32c06c8f0 0x7fe32c06edb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 --2- 172.21.15.70:0/765411052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe340067c60 0x7fe3401e54d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 >> 172.21.15.70:0/765411052 conn(0x7fe340073360 msgr2=0x7fe34006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:14.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 shutdown_connections 2024-04-15T15:16:14.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.214+0000 7fe346863640 1 -- 172.21.15.70:0/765411052 wait complete. 2024-04-15T15:16:14.228 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:16:14.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.332+0000 7ff893ddb640 1 Processor -- start 2024-04-15T15:16:14.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.332+0000 7ff893ddb640 1 -- start start 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff893ddb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff88c15ec10 0x7ff88c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff893ddb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff88c0b53c0 0x7ff88c0b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff893ddb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff88c067e30 con 0x7ff88c15e830 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff893ddb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c0b3c10 con 0x7ff88c0b53c0 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff893ddb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff88c0b3ef0 con 0x7ff88c15ec10 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff88c0b53c0 0x7ff88c0b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff88c0b53c0 0x7ff88c0b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44864/0 (socket says 172.21.15.70:44864) 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 -- 172.21.15.70:0/2976252136 learned_addr learned my addr 172.21.15.70:0/2976252136 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff891376640 1 --2- 172.21.15.70:0/2976252136 >> v2:172.21.15.96:3300/0 conn(0x7ff88c15ec10 0x7ff88c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 -- 172.21.15.70:0/2976252136 >> v2:172.21.15.96:3300/0 conn(0x7ff88c15ec10 msgr2=0x7ff88c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 --2- 172.21.15.70:0/2976252136 >> v2:172.21.15.96:3300/0 conn(0x7ff88c15ec10 0x7ff88c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.333+0000 7ff892378640 1 -- 172.21.15.70:0/2976252136 >> v1:172.21.15.96:6789/0 conn(0x7ff88c15e830 legacy=0x7ff88c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:14.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.334+0000 7ff892378640 1 -- 172.21.15.70:0/2976252136 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c0b41d0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.334+0000 7ff892378640 1 --2- 172.21.15.70:0/2976252136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c0b5870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06e050 tx=0x7ff88402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a9755f280c2f43 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.334+0000 7ff890b75640 1 -- 172.21.15.70:0/2976252136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884035020 con 0x7ff88c0b53c0 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.334+0000 7ff890b75640 1 -- 172.21.15.70:0/2976252136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff884031870 con 0x7ff88c0b53c0 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.334+0000 7ff890b75640 1 -- 172.21.15.70:0/2976252136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff884031c40 con 0x7ff88c0b53c0 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 -- 172.21.15.70:0/2976252136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 msgr2=0x7ff88c0b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 --2- 172.21.15.70:0/2976252136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c0b5870 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7ff88c06e050 tx=0x7ff88402b910 comp rx=0 tx=0).stop 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 -- 172.21.15.70:0/2976252136 shutdown_connections 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 --2- 172.21.15.70:0/2976252136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c0b5870 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 --2- 172.21.15.70:0/2976252136 >> v2:172.21.15.96:3300/0 conn(0x7ff88c15ec10 0x7ff88c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 -- 172.21.15.70:0/2976252136 >> 172.21.15.70:0/2976252136 conn(0x7ff88c073360 msgr2=0x7ff88c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:14.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 -- 172.21.15.70:0/2976252136 shutdown_connections 2024-04-15T15:16:14.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.335+0000 7ff893ddb640 1 -- 172.21.15.70:0/2976252136 wait complete. 2024-04-15T15:16:14.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.336+0000 7ff893ddb640 1 Processor -- start 2024-04-15T15:16:14.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.336+0000 7ff893ddb640 1 -- start start 2024-04-15T15:16:14.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff88c15e830 0x7ff88c14c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff88c0c5ae0 0x7ff88c15d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c15ff60 con 0x7ff88c0b53c0 2024-04-15T15:16:14.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff88c0b4790 con 0x7ff88c0c5ae0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff893ddb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff88c1600e0 con 0x7ff88c15e830 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44866/0 (socket says 172.21.15.70:44866) 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 -- 172.21.15.70:0/3435721789 learned_addr learned my addr 172.21.15.70:0/3435721789 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891376640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff88c15e830 0x7ff88c14c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff892378640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff88c0c5ae0 0x7ff88c15d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 -- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff88c15e830 msgr2=0x7ff88c14c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff88c15e830 0x7ff88c14c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 -- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff88c0c5ae0 msgr2=0x7ff88c15d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff88c0c5ae0 0x7ff88c15d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 -- 172.21.15.70:0/3435721789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff88c1e7b40 con 0x7ff88c0b53c0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.337+0000 7ff891b77640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c7c0 tx=0x7ff88800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.338+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888016020 con 0x7ff88c0b53c0 2024-04-15T15:16:14.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.338+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff888006d50 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.338+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff88c1e7e20 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.338+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888003bb0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.338+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff88c1e86c0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.339+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff88c06d1d0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.339+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7ff88800a850 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.340+0000 7ff8827fc640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 0x7ff87006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.340+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff88808bae0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.340+0000 7ff891376640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 0x7ff87006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.340+0000 7ff891376640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 0x7ff87006edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15ddf0 tx=0x7ff87c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.344+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff88c06d1d0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.558+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7ff88c0ae3f0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.560 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-15T15:16:14.559+0000 7fdd5b7bd640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:16:14.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.560+0000 7ff8827fc640 1 -- 172.21.15.70:0/3435721789 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7ff88c0ae3f0 con 0x7ff88c0b53c0 2024-04-15T15:16:14.562 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 msgr2=0x7ff87006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 0x7ff87006edb0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff88c15ddf0 tx=0x7ff87c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 msgr2=0x7ff88c1e8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff88800c7c0 tx=0x7ff88800cc80 comp rx=0 tx=0).stop 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 shutdown_connections 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff88c0c5ae0 0x7ff88c15d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff88c15e830 0x7ff88c14c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff87006c8f0 0x7ff87006edb0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 --2- 172.21.15.70:0/3435721789 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff88c0b53c0 0x7ff88c1e8be0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 >> 172.21.15.70:0/3435721789 conn(0x7ff88c073360 msgr2=0x7ff88c0af650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:14.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 shutdown_connections 2024-04-15T15:16:14.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.563+0000 7ff893ddb640 1 -- 172.21.15.70:0/3435721789 wait complete. 2024-04-15T15:16:14.566 INFO:tasks.ceph.mds.c.smithi070.stderr: -5765> 2024-04-15T15:15:03.624+0000 7f5b2b627640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-04-15T15:16:14.566 INFO:tasks.ceph.mds.c.smithi070.stderr: -4476> 2024-04-15T15:15:14.635+0000 7f5b2561b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:14.577 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:16:14.597 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-15T15:16:14.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.682+0000 7f40fc44a640 1 Processor -- start 2024-04-15T15:16:14.684 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.682+0000 7f40fc44a640 1 -- start start 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.683+0000 7f40fc44a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40f40d3270 0x7f40f40cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.683+0000 7f40fc44a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f40b47a0 con 0x7f40f415e700 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.683+0000 7f40fc44a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f40b4a80 con 0x7f40f40d2360 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.683+0000 7f40fc44a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f40f40b53b0 con 0x7f40f40d3270 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa1e6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f40f415e700 0x7f40f40b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54588/0 (socket says 172.21.15.70:54588) 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa1e6640 1 -- 172.21.15.70:0/318478926 learned_addr learned my addr 172.21.15.70:0/318478926 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa9e7640 1 --2- 172.21.15.70:0/318478926 >> v2:172.21.15.96:3300/0 conn(0x7f40f40d3270 0x7f40f40cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40f91e4640 1 -- 172.21.15.70:0/318478926 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 813969748 0 0) 0x7f40f40b47a0 con 0x7f40f415e700 2024-04-15T15:16:14.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40f91e4640 1 -- 172.21.15.70:0/318478926 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40e0003520 con 0x7f40f415e700 2024-04-15T15:16:14.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa9e7640 1 -- 172.21.15.70:0/318478926 >> v1:172.21.15.96:6789/0 conn(0x7f40f40d2360 legacy=0x7f40f40d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa9e7640 1 -- 172.21.15.70:0/318478926 >> v1:172.21.15.70:6789/0 conn(0x7f40f415e700 legacy=0x7f40f40b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.684+0000 7f40fa9e7640 1 -- 172.21.15.70:0/318478926 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f40b4d60 con 0x7f40f40d3270 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.685+0000 7f40fa9e7640 1 --2- 172.21.15.70:0/318478926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f40d3270 0x7f40f40cf370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40f40b5530 tx=0x7f40ec02bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8080a5ccfa35fd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.685+0000 7f40f91e4640 1 -- 172.21.15.70:0/318478926 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec039070 con 0x7f40f40d3270 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.685+0000 7f40f91e4640 1 -- 172.21.15.70:0/318478926 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ec02cce0 con 0x7f40f40d3270 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.685+0000 7f40f91e4640 1 -- 172.21.15.70:0/318478926 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ec034640 con 0x7f40f40d3270 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 -- 172.21.15.70:0/318478926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f40d3270 msgr2=0x7f40f40cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 --2- 172.21.15.70:0/318478926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f40d3270 0x7f40f40cf370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f40f40b5530 tx=0x7f40ec02bf30 comp rx=0 tx=0).stop 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 -- 172.21.15.70:0/318478926 shutdown_connections 2024-04-15T15:16:14.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 --2- 172.21.15.70:0/318478926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f40d3270 0x7f40f40cf370 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 -- 172.21.15.70:0/318478926 >> 172.21.15.70:0/318478926 conn(0x7f40f408bf60 msgr2=0x7f40f40d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:14.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 -- 172.21.15.70:0/318478926 shutdown_connections 2024-04-15T15:16:14.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.686+0000 7f40fc44a640 1 -- 172.21.15.70:0/318478926 wait complete. 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.687+0000 7f40fc44a640 1 Processor -- start 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.687+0000 7f40fc44a640 1 -- start start 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f40d2360 0x7f40f4150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 0x7f40f409f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40f40d1d40 con 0x7f40f40d3270 2024-04-15T15:16:14.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40f40b4450 con 0x7f40f415e700 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f40f40d1ec0 con 0x7f40f40d2360 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44890/0 (socket says 172.21.15.70:44890) 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 -- 172.21.15.70:0/436628295 learned_addr learned my addr 172.21.15.70:0/436628295 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fa1e6640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f40d2360 0x7f40f4150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fa9e7640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 0x7f40f409f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 -- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f40d2360 msgr2=0x7f40f4150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f40d2360 0x7f40f4150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 -- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 msgr2=0x7f40f409f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 0x7f40f409f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 -- 172.21.15.70:0/436628295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f40b7af0 con 0x7f40f40d3270 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fa9e7640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 0x7f40f409f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40f99e5640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f40f000c400 tx=0x7f40f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f00052d0 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f0005490 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f40b7dd0 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.688+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0007260 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.689+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f40b8390 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.689+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f40689e0 con 0x7f40f40d3270 2024-04-15T15:16:14.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.690+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40f0007400 con 0x7f40f40d3270 2024-04-15T15:16:14.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.690+0000 7f40e6ffd640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 0x7f40d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:14.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.690+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f40f008b520 con 0x7f40f40d3270 2024-04-15T15:16:14.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.690+0000 7f40fa1e6640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 0x7f40d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:14.694 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.691+0000 7f40fa1e6640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 0x7f40d806edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f40e80010e0 tx=0x7f40e8007be0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:14.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.695+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40f0058e10 con 0x7f40f40d3270 2024-04-15T15:16:14.909 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:14.907+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f40f40ba500 con 0x7f40f40d3270 2024-04-15T15:16:15.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.561+0000 7f40e6ffd640 1 -- 172.21.15.70:0/436628295 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f40f005d110 con 0x7f40f40d3270 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 msgr2=0x7f40d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 0x7f40d806edb0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f40e80010e0 tx=0x7f40e8007be0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 msgr2=0x7f40f40b7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f40f000c400 tx=0x7f40f000c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 shutdown_connections 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f415e700 0x7f40f409f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d806c8f0 0x7f40d806edb0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f40d3270 0x7f40f40b7180 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 --2- 172.21.15.70:0/436628295 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f40d2360 0x7f40f4150c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 >> 172.21.15.70:0/436628295 conn(0x7f40f408bf60 msgr2=0x7f40f40d2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 shutdown_connections 2024-04-15T15:16:15.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.564+0000 7f40fc44a640 1 -- 172.21.15.70:0/436628295 wait complete. 2024-04-15T15:16:15.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:16:15.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.726+0000 7f7b35be8640 1 Processor -- start 2024-04-15T15:16:15.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.726+0000 7f7b35be8640 1 -- start start 2024-04-15T15:16:15.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b300b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b300b1520 0x7f7b3009b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7b301626e0 0x7f7b30162b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f7b30163070 con 0x7f7b300b1520 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f7b300d0d90 con 0x7f7b301626e0 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b35be8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7b300b1a20 con 0x7f7b3015eb70 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b300b1520 0x7f7b3009b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7b300b1520 0x7f7b3009b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44902/0 (socket says 172.21.15.70:44902) 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 -- 172.21.15.70:0/4260460519 learned_addr learned my addr 172.21.15.70:0/4260460519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2ffff640 1 --2- 172.21.15.70:0/4260460519 >> v2:172.21.15.96:3300/0 conn(0x7f7b301626e0 0x7f7b30162b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2f7fe640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b300b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 -- 172.21.15.70:0/4260460519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 msgr2=0x7f7b300b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b300b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 -- 172.21.15.70:0/4260460519 >> v2:172.21.15.96:3300/0 conn(0x7f7b301626e0 msgr2=0x7f7b30162b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 --2- 172.21.15.70:0/4260460519 >> v2:172.21.15.96:3300/0 conn(0x7f7b301626e0 0x7f7b30162b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.727+0000 7f7b2effd640 1 -- 172.21.15.70:0/4260460519 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b300d0fe0 con 0x7f7b300b1520 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.728+0000 7f7b2effd640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b3009b7c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b30073690 tx=0x7f7b2402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88a77b5b34b8718 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.728+0000 7f7b2e7fc640 1 -- 172.21.15.70:0/4260460519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2402e070 con 0x7f7b300b1520 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.728+0000 7f7b2e7fc640 1 -- 172.21.15.70:0/4260460519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b24037070 con 0x7f7b300b1520 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.728+0000 7f7b2e7fc640 1 -- 172.21.15.70:0/4260460519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b24031a00 con 0x7f7b300b1520 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 -- 172.21.15.70:0/4260460519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 msgr2=0x7f7b3009b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b3009b7c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7b30073690 tx=0x7f7b2402b880 comp rx=0 tx=0).stop 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 -- 172.21.15.70:0/4260460519 shutdown_connections 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 --2- 172.21.15.70:0/4260460519 >> v2:172.21.15.96:3300/0 conn(0x7f7b301626e0 0x7f7b30162b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b3009b7c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 --2- 172.21.15.70:0/4260460519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b300b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 -- 172.21.15.70:0/4260460519 >> 172.21.15.70:0/4260460519 conn(0x7f7b300b3f70 msgr2=0x7f7b300b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 -- 172.21.15.70:0/4260460519 shutdown_connections 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.729+0000 7f7b35be8640 1 -- 172.21.15.70:0/4260460519 wait complete. 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.730+0000 7f7b35be8640 1 Processor -- start 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.730+0000 7f7b35be8640 1 -- start start 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b3015ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 0x7f7b300c9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7b3009bd00 con 0x7f7b300b1520 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7b301634a0 con 0x7f7b301626e0 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b35be8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f7b3006d160 con 0x7f7b3015eb70 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44908/0 (socket says 172.21.15.70:44908) 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 -- 172.21.15.70:0/1769813542 learned_addr learned my addr 172.21.15.70:0/1769813542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:15.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2effd640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b3015ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2ffff640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 0x7f7b300c9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 -- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 msgr2=0x7f7b3015ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b3015ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 -- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 msgr2=0x7f7b300c9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 0x7f7b300c9550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 -- 172.21.15.70:0/1769813542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b30162370 con 0x7f7b300b1520 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2ffff640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 0x7f7b300c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.731+0000 7f7b2f7fe640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007e60 tx=0x7f7b2000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.732+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20014620 con 0x7f7b300b1520 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.732+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b20019070 con 0x7f7b300b1520 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.732+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b3015c3b0 con 0x7f7b300b1520 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.732+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20014a00 con 0x7f7b300b1520 2024-04-15T15:16:15.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.732+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b300cc7c0 con 0x7f7b300b1520 2024-04-15T15:16:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.733+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b300689e0 con 0x7f7b300b1520 2024-04-15T15:16:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.733+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f7b20014ba0 con 0x7f7b300b1520 2024-04-15T15:16:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.733+0000 7f7b17fff640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 0x7f7b1c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.734+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f7b2008c700 con 0x7f7b300b1520 2024-04-15T15:16:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.734+0000 7f7b2effd640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 0x7f7b1c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.734+0000 7f7b2effd640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 0x7f7b1c06edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7b24036000 tx=0x7f7b2403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.738+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7b2005a760 con 0x7f7b300b1520 2024-04-15T15:16:15.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:15.914+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f7b300b8400 con 0x7f7b300b1520 2024-04-15T15:16:16.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.567+0000 7f7b17fff640 1 -- 172.21.15.70:0/1769813542 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7f7b2005a060 con 0x7f7b300b1520 2024-04-15T15:16:16.569 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 msgr2=0x7f7b1c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 0x7f7b1c06edb0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7b24036000 tx=0x7f7b2403d040 comp rx=0 tx=0).stop 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 msgr2=0x7f7b30161d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7b20007e60 tx=0x7f7b2000e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 shutdown_connections 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7b301626e0 0x7f7b300c9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7b3015eb70 0x7f7b3015ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f7b1c06c8f0 0x7f7b1c06edb0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 --2- 172.21.15.70:0/1769813542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7b300b1520 0x7f7b30161d00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 >> 172.21.15.70:0/1769813542 conn(0x7f7b300b3f70 msgr2=0x7f7b3015ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 shutdown_connections 2024-04-15T15:16:16.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.570+0000 7f7b35be8640 1 -- 172.21.15.70:0/1769813542 wait complete. 2024-04-15T15:16:16.584 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:16:16.695 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 Processor -- start 2024-04-15T15:16:16.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 -- start start 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8f815e8c0 0x7fa8f80d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 0x7fa8f80d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8f80d2340 con 0x7fa8f80d0140 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f815eca0 con 0x7fa8f815e8c0 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.694+0000 7fa8fe196640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa8f80d2620 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8f815e8c0 0x7fa8f80d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa8f815e8c0 0x7fa8f80d3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44920/0 (socket says 172.21.15.70:44920) 2024-04-15T15:16:16.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 -- 172.21.15.70:0/1724818499 learned_addr learned my addr 172.21.15.70:0/1724818499 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 -- 172.21.15.70:0/1724818499 >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 msgr2=0x7fa8f80d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f6ffd640 1 --2- 172.21.15.70:0/1724818499 >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 0x7fa8f80d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 --2- 172.21.15.70:0/1724818499 >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 0x7fa8f80d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 -- 172.21.15.70:0/1724818499 >> v1:172.21.15.96:6789/0 conn(0x7fa8f80d0140 legacy=0x7fa8f806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f77fe640 1 -- 172.21.15.70:0/1724818499 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f80d2900 con 0x7fa8f815e8c0 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.695+0000 7fa8f6ffd640 1 --2- 172.21.15.70:0/1724818499 >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 0x7fa8f80d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:16.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8f77fe640 1 --2- 172.21.15.70:0/1724818499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80d3100 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80d1e90 tx=0x7fa8e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bbcada21165bd2c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8f67fc640 1 -- 172.21.15.70:0/1724818499 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e802e070 con 0x7fa8f815e8c0 2024-04-15T15:16:16.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8f67fc640 1 -- 172.21.15.70:0/1724818499 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8037070 con 0x7fa8f815e8c0 2024-04-15T15:16:16.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8f67fc640 1 -- 172.21.15.70:0/1724818499 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8031a00 con 0x7fa8f815e8c0 2024-04-15T15:16:16.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 -- 172.21.15.70:0/1724818499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f815e8c0 msgr2=0x7fa8f80d3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 --2- 172.21.15.70:0/1724818499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80d3100 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80d1e90 tx=0x7fa8e802b880 comp rx=0 tx=0).stop 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 -- 172.21.15.70:0/1724818499 shutdown_connections 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 --2- 172.21.15.70:0/1724818499 >> v2:172.21.15.96:3300/0 conn(0x7fa8f80cfcb0 0x7fa8f80d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 --2- 172.21.15.70:0/1724818499 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80d3100 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 -- 172.21.15.70:0/1724818499 >> 172.21.15.70:0/1724818499 conn(0x7fa8f808bf60 msgr2=0x7fa8f80d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 -- 172.21.15.70:0/1724818499 shutdown_connections 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.696+0000 7fa8fe196640 1 -- 172.21.15.70:0/1724818499 wait complete. 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.697+0000 7fa8fe196640 1 Processor -- start 2024-04-15T15:16:16.700 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 -- start start 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8f80d0140 0x7fa8f80b87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80b8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f80d4f60 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa8f80d4de0 con 0x7fa8f815e8c0 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8fe196640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa8f80d13c0 con 0x7fa8f80d0140 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.698+0000 7fa8f77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44924/0 (socket says 172.21.15.70:44924) 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 -- 172.21.15.70:0/624688050 learned_addr learned my addr 172.21.15.70:0/624688050 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:16.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 -- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8f80d0140 msgr2=0x7fa8f80b87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8f80d0140 0x7fa8f80b87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 -- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8f815e8c0 msgr2=0x7fa8f80b8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80b8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 -- 172.21.15.70:0/624688050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8f80cc4d0 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8f77fe640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802b8c0 tx=0x7fa8e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8032880 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8e8032a20 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8f80cc7b0 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8e8032e10 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.699+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8f80ccd70 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.703 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.700+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8f80ccfb0 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.701+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa8e804c060 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.701+0000 7fa8effff640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 0x7fa8d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:16.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.701+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7fa8e80ad980 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.701+0000 7fa8f6ffd640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 0x7fa8d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:16.704 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.702+0000 7fa8f6ffd640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 0x7fa8d806edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0006fd0 tx=0x7fa8e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:16.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.706+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa8f80ccfb0 con 0x7fa8f80cfcb0 2024-04-15T15:16:16.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:16.882+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa8f80689e0 con 0x7fa8f80cfcb0 2024-04-15T15:16:17.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.573+0000 7fa8effff640 1 -- 172.21.15.70:0/624688050 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7fa8e8040070 con 0x7fa8f80cfcb0 2024-04-15T15:16:17.575 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-15T15:16:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 msgr2=0x7fa8d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 0x7fa8d806edb0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa8e0006fd0 tx=0x7fa8e0008040 comp rx=0 tx=0).stop 2024-04-15T15:16:17.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 msgr2=0x7fa8f8152cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802b8c0 tx=0x7fa8e80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 shutdown_connections 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa8f815e8c0 0x7fa8f80b8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa8f80d0140 0x7fa8f80b87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa8d806c8f0 0x7fa8d806edb0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 --2- 172.21.15.70:0/624688050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa8f80cfcb0 0x7fa8f8152cc0 secure :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa8e802b8c0 tx=0x7fa8e80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.576+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 >> 172.21.15.70:0/624688050 conn(0x7fa8f808bf60 msgr2=0x7fa8f80b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.577+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 shutdown_connections 2024-04-15T15:16:17.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.577+0000 7fa8fe196640 1 -- 172.21.15.70:0/624688050 wait complete. 2024-04-15T15:16:17.593 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-15T15:16:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.703+0000 7eff4482a640 1 Processor -- start 2024-04-15T15:16:17.705 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.703+0000 7eff4482a640 1 -- start start 2024-04-15T15:16:17.706 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff4482a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 0x7eff4009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff4482a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7eff400d17b0 0x7eff4006d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff4482a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff400d09f0 con 0x7eff4009b920 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff4482a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7eff400d1320 con 0x7eff400d17b0 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff4482a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7eff400d0cd0 con 0x7eff4015e700 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff37fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7eff400d17b0 0x7eff4006d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff3ed9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7eff4009b920 0x7eff400cdc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54592/0 (socket says 172.21.15.70:54592) 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff3ed9d640 1 -- 172.21.15.70:0/4022862876 learned_addr learned my addr 172.21.15.70:0/4022862876 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff3e59c640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 0x7eff4009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff37fff640 1 -- 172.21.15.70:0/4022862876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 msgr2=0x7eff4009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff37fff640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 0x7eff4009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff37fff640 1 -- 172.21.15.70:0/4022862876 >> v1:172.21.15.70:6789/0 conn(0x7eff4009b920 legacy=0x7eff400cdc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff37fff640 1 -- 172.21.15.70:0/4022862876 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff400d0fb0 con 0x7eff400d17b0 2024-04-15T15:16:17.707 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.704+0000 7eff3e59c640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 0x7eff4009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.705+0000 7eff37fff640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff400d17b0 0x7eff4006d500 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff400d14a0 tx=0x7eff2802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998f7717cd6c1dff server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.705+0000 7eff3dd9b640 1 -- 172.21.15.70:0/4022862876 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28035020 con 0x7eff400d17b0 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.705+0000 7eff3dd9b640 1 -- 172.21.15.70:0/4022862876 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff28031800 con 0x7eff400d17b0 2024-04-15T15:16:17.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.705+0000 7eff3dd9b640 1 -- 172.21.15.70:0/4022862876 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff28031bd0 con 0x7eff400d17b0 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 -- 172.21.15.70:0/4022862876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff400d17b0 msgr2=0x7eff4006d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff400d17b0 0x7eff4006d500 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7eff400d14a0 tx=0x7eff2802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 -- 172.21.15.70:0/4022862876 shutdown_connections 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff400d17b0 0x7eff4006d500 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 --2- 172.21.15.70:0/4022862876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff4015e700 0x7eff4009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 -- 172.21.15.70:0/4022862876 >> 172.21.15.70:0/4022862876 conn(0x7eff400b3ec0 msgr2=0x7eff400b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 -- 172.21.15.70:0/4022862876 shutdown_connections 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.706+0000 7eff4482a640 1 -- 172.21.15.70:0/4022862876 wait complete. 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.707+0000 7eff4482a640 1 Processor -- start 2024-04-15T15:16:17.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.707+0000 7eff4482a640 1 -- start start 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff4009b920 0x7eff40161070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 0x7eff401507e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7eff400d0790 con 0x7eff4015e700 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7eff400d0610 con 0x7eff4009b920 2024-04-15T15:16:17.711 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff4482a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7eff4015eae0 con 0x7eff400d17b0 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44942/0 (socket says 172.21.15.70:44942) 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 -- 172.21.15.70:0/1071925352 learned_addr learned my addr 172.21.15.70:0/1071925352 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff37fff640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 0x7eff401507e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 -- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 msgr2=0x7eff401507e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 0x7eff401507e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 -- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff4009b920 msgr2=0x7eff40161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff4009b920 0x7eff40161070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 -- 172.21.15.70:0/1071925352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff401615b0 con 0x7eff4015e700 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff37fff640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 0x7eff401507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff3ed9d640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff3800ca80 tx=0x7eff3800cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38017070 con 0x7eff4015e700 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.708+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff3800ac30 con 0x7eff4015e700 2024-04-15T15:16:17.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.709+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff40160860 con 0x7eff4015e700 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.709+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff38012600 con 0x7eff4015e700 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.709+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff4015efe0 con 0x7eff4015e700 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.710+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff400690f0 con 0x7eff4015e700 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.710+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7eff3801a070 con 0x7eff4015e700 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.710+0000 7eff36ffd640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 0x7eff2006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:17.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.711+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7eff3808b700 con 0x7eff4015e700 2024-04-15T15:16:17.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.711+0000 7eff3e59c640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 0x7eff2006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:17.714 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.711+0000 7eff3e59c640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 0x7eff2006ede0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff30006fd0 tx=0x7eff30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:17.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.715+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7eff401615b0 con 0x7eff4015e700 2024-04-15T15:16:17.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:17.896+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7eff400d2130 con 0x7eff4015e700 2024-04-15T15:16:18.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.578+0000 7eff36ffd640 1 -- 172.21.15.70:0/1071925352 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7eff3800ae20 con 0x7eff4015e700 2024-04-15T15:16:18.580 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data_ec' removed 2024-04-15T15:16:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 msgr2=0x7eff2006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 0x7eff2006ede0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7eff30006fd0 tx=0x7eff30008040 comp rx=0 tx=0).stop 2024-04-15T15:16:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 msgr2=0x7eff4015f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7eff3800ca80 tx=0x7eff3800cf40 comp rx=0 tx=0).stop 2024-04-15T15:16:18.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 shutdown_connections 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7eff2006c920 0x7eff2006ede0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7eff4015e700 0x7eff4015f7e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7eff400d17b0 0x7eff401507e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 --2- 172.21.15.70:0/1071925352 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7eff4009b920 0x7eff40161070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.581+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 >> 172.21.15.70:0/1071925352 conn(0x7eff400b3ec0 msgr2=0x7eff40070cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.582+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 shutdown_connections 2024-04-15T15:16:18.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.582+0000 7eff4482a640 1 -- 172.21.15.70:0/1071925352 wait complete. 2024-04-15T15:16:18.596 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-15T15:16:18.596 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-15T15:16:18.596 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:16:18.596 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:18.596 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-15T15:16:18.596 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-15T15:16:18.597 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:16:18.597 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:18.597 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-15T15:16:18.597 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-15T15:16:18.597 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:16:18.597 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:18.597 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-15T15:16:18.598 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-15T15:16:18.598 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:16:18.598 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:18.607 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:16:18.607 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-15T15:16:18.608 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:16:18.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-15T15:16:18.615 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:16:18.615 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-15T15:16:18.616 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:16:18.616 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-15T15:16:18.650 INFO:tasks.ceph.mds.a:Started 2024-04-15T15:16:18.650 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:18.651 INFO:tasks.ceph.mds.c:Started 2024-04-15T15:16:18.651 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:18.658 INFO:tasks.ceph.mds.d:Started 2024-04-15T15:16:18.659 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:18.659 INFO:tasks.ceph.mds.b:Started 2024-04-15T15:16:18.660 DEBUG:teuthology.parallel:result is None 2024-04-15T15:16:18.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-15T15:16:18.730 INFO:tasks.ceph.mds.c.smithi070.stdout:starting mds.c at 2024-04-15T15:16:18.730 INFO:tasks.ceph.mds.a.smithi070.stdout:starting mds.a at 2024-04-15T15:16:18.738 INFO:tasks.ceph.mds.d.smithi096.stdout:starting mds.d at 2024-04-15T15:16:18.739 INFO:tasks.ceph.mds.b.smithi096.stdout:starting mds.b at 2024-04-15T15:16:18.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.767+0000 7f2cf73ba640 1 Processor -- start 2024-04-15T15:16:18.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.768+0000 7f2cf73ba640 1 -- start start 2024-04-15T15:16:18.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf73ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cf015e700 0x7f2cf00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf73ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 0x7f2cf00cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf73ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf00b47a0 con 0x7f2cf00d2360 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf73ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf00b53b0 con 0x7f2cf015e700 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf73ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00b4a80 con 0x7f2cf00d3270 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cf015e700 0x7f2cf00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2cf015e700 0x7f2cf00b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44988/0 (socket says 172.21.15.70:44988) 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 -- 172.21.15.70:0/4039822492 learned_addr learned my addr 172.21.15.70:0/4039822492 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5957640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 0x7f2cf00cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 -- 172.21.15.70:0/4039822492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 msgr2=0x7f2cf00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 0x7f2cf00cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 -- 172.21.15.70:0/4039822492 >> v1:172.21.15.96:6789/0 conn(0x7f2cf00d2360 legacy=0x7f2cf00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5156640 1 -- 172.21.15.70:0/4039822492 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf00b4d60 con 0x7f2cf015e700 2024-04-15T15:16:18.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.769+0000 7f2cf5957640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 0x7f2cf00cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2cf5156640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf015e700 0x7f2cf00b5840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009850 tx=0x7f2ce002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23851854dfeceda8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2ce7fff640 1 -- 172.21.15.70:0/4039822492 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0004230 con 0x7f2cf015e700 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2ce7fff640 1 -- 172.21.15.70:0/4039822492 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce00043d0 con 0x7f2cf015e700 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2ce7fff640 1 -- 172.21.15.70:0/4039822492 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0003810 con 0x7f2cf015e700 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2cf73ba640 1 -- 172.21.15.70:0/4039822492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf015e700 msgr2=0x7f2cf00b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.770+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf015e700 0x7f2cf00b5840 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009850 tx=0x7f2ce002b960 comp rx=0 tx=0).stop 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 -- 172.21.15.70:0/4039822492 shutdown_connections 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d3270 0x7f2cf00cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/4039822492 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf015e700 0x7f2cf00b5840 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 -- 172.21.15.70:0/4039822492 >> 172.21.15.70:0/4039822492 conn(0x7f2cf008bf60 msgr2=0x7f2cf00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 -- 172.21.15.70:0/4039822492 shutdown_connections 2024-04-15T15:16:18.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.771+0000 7f2cf73ba640 1 -- 172.21.15.70:0/4039822492 wait complete. 2024-04-15T15:16:18.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.772+0000 7f2cf73ba640 1 Processor -- start 2024-04-15T15:16:18.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.772+0000 7f2cf73ba640 1 -- start start 2024-04-15T15:16:18.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 0x7f2cf015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 0x7f2cf015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00b5d80 con 0x7f2cf00d3270 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00b4450 con 0x7f2cf015e700 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf00d1d40 con 0x7f2cf00d2360 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:44992/0 (socket says 172.21.15.70:44992) 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 -- 172.21.15.70:0/787575679 learned_addr learned my addr 172.21.15.70:0/787575679 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf5156640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 0x7f2cf015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf5957640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 0x7f2cf015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 -- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 msgr2=0x7f2cf015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 0x7f2cf015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 -- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 msgr2=0x7f2cf015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:18.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 0x7f2cf015eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 -- 172.21.15.70:0/787575679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf00cc700 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf5156640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 0x7f2cf015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf5957640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 0x7f2cf015eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf4955640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c480 tx=0x7f2cec00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0052d0 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cec005470 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf00cc9e0 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.773+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cec012870 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.774+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf00ccfa0 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.774+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf00689e0 con 0x7f2cf00d3270 2024-04-15T15:16:18.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.775+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2cec012a10 con 0x7f2cf00d3270 2024-04-15T15:16:18.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.776+0000 7f2ce5ffb640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 0x7f2cd406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:18.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.776+0000 7f2cf5156640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 0x7f2cd406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:18.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.776+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f2cec08b5f0 con 0x7f2cf00d3270 2024-04-15T15:16:18.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.776+0000 7f2cf5156640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 0x7f2cd406edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0035000 tx=0x7f2ce0008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:18.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.780+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2cec059c20 con 0x7f2cf00d3270 2024-04-15T15:16:18.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:18.971+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f2cf00ba500 con 0x7f2cf00d3270 2024-04-15T15:16:19.592 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.590+0000 7f2ce5ffb640 1 -- 172.21.15.70:0/787575679 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f2cec059520 con 0x7f2cf00d3270 2024-04-15T15:16:19.592 INFO:teuthology.orchestra.run.smithi070.stderr: removed all blocklist entries 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 msgr2=0x7f2cd406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 0x7f2cd406edb0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0035000 tx=0x7f2ce0008030 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 msgr2=0x7f2cf0161610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2cec00c480 tx=0x7f2cec00c940 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 shutdown_connections 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2cf015e700 0x7f2cf015eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2cd406c8f0 0x7f2cd406edb0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2cf00d3270 0x7f2cf0161610 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 --2- 172.21.15.70:0/787575679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2cf00d2360 0x7f2cf015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 >> 172.21.15.70:0/787575679 conn(0x7f2cf008bf60 msgr2=0x7f2cf00d0b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:19.595 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 shutdown_connections 2024-04-15T15:16:19.596 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.593+0000 7f2cf73ba640 1 -- 172.21.15.70:0/787575679 wait complete. 2024-04-15T15:16:19.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-15T15:16:19.715 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.714+0000 7f3d62642640 1 Processor -- start 2024-04-15T15:16:19.716 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.714+0000 7f3d62642640 1 -- start start 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.714+0000 7f3d62642640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d5c15e830 0x7f3d5c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d62642640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5c067e30 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d62642640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5c0b3c10 con 0x7f3d5c15ec10 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d62642640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5c0b3ef0 con 0x7f3d5c15e830 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d5bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3d5c15e830 0x7f3d5c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d60bdf640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3d5c0b53c0 0x7f3d5c0aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54614/0 (socket says 172.21.15.70:54614) 2024-04-15T15:16:19.717 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d60bdf640 1 -- 172.21.15.70:0/4086818766 learned_addr learned my addr 172.21.15.70:0/4086818766 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d5bfff640 1 -- 172.21.15.70:0/4086818766 >> v1:172.21.15.96:6789/0 conn(0x7f3d5c15ec10 legacy=0x7f3d5c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d5bfff640 1 -- 172.21.15.70:0/4086818766 >> v1:172.21.15.70:6789/0 conn(0x7f3d5c0b53c0 legacy=0x7f3d5c0aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d5bfff640 1 -- 172.21.15.70:0/4086818766 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c0b41d0 con 0x7f3d5c15e830 2024-04-15T15:16:19.718 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.715+0000 7f3d5affd640 1 -- 172.21.15.70:0/4086818766 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 916741554 0 0) 0x7f3d5c067e30 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d5bfff640 1 --2- 172.21.15.70:0/4086818766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0b4cb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b19e8e8b7d852d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d5affd640 1 -- 172.21.15.70:0/4086818766 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c0315a0 con 0x7f3d5c15e830 2024-04-15T15:16:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d5affd640 1 -- 172.21.15.70:0/4086818766 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4c037070 con 0x7f3d5c15e830 2024-04-15T15:16:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d5affd640 1 -- 172.21.15.70:0/4086818766 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c031960 con 0x7f3d5c15e830 2024-04-15T15:16:19.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d62642640 1 -- 172.21.15.70:0/4086818766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c15e830 msgr2=0x7f3d5c0b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d62642640 1 --2- 172.21.15.70:0/4086818766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0b4cb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c02b840 comp rx=0 tx=0).stop 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d62642640 1 -- 172.21.15.70:0/4086818766 shutdown_connections 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d62642640 1 --2- 172.21.15.70:0/4086818766 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0b4cb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.716+0000 7f3d62642640 1 -- 172.21.15.70:0/4086818766 >> 172.21.15.70:0/4086818766 conn(0x7f3d5c073360 msgr2=0x7f3d5c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.717+0000 7f3d62642640 1 -- 172.21.15.70:0/4086818766 shutdown_connections 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.717+0000 7f3d62642640 1 -- 172.21.15.70:0/4086818766 wait complete. 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.718+0000 7f3d62642640 1 Processor -- start 2024-04-15T15:16:19.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.718+0000 7f3d62642640 1 -- start start 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.718+0000 7f3d62642640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.718+0000 7f3d62642640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.718+0000 7f3d62642640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d5c14f420 0x7f3d5c15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d62642640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c15ff60 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d62642640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5c0b4790 con 0x7f3d5c15e830 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d62642640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5c1600e0 con 0x7f3d5c14f420 2024-04-15T15:16:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45018/0 (socket says 172.21.15.70:45018) 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 -- 172.21.15.70:0/2939635048 learned_addr learned my addr 172.21.15.70:0/2939635048 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 -- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d5c14f420 msgr2=0x7f3d5c15cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d5c14f420 0x7f3d5c15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 -- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c15e830 msgr2=0x7f3d5c0c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 -- 172.21.15.70:0/2939635048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5c0c6250 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d5bfff640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c004030 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d4c0041f0 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5c0c6530 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d4c032640 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.719+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5c0c6af0 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.723 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.720+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d5c0689e0 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.721+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3d4c0327e0 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.721+0000 7f3d58ff9640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 0x7f3d4806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.721+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f3d4c02f030 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.721+0000 7f3d5b7fe640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 0x7f3d4806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.721+0000 7f3d5b7fe640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 0x7f3d4806edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c15e1e0 tx=0x7f3d500079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.726+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3d4c041070 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.968+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f3d5c06ff10 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.969+0000 7f3d58ff9640 1 -- 172.21.15.70:0/2939635048 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f3d4c07c080 con 0x7f3d5c0b53c0 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "auth_dump": [ 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.a", 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBFRB1mKChoNRAAFnzGTfpU/J9EU9lWKg7E8w==", 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.971 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.b", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1m8c2qBhAAB8LIa881VpeaEHNlYqR7RA==", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.972 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.c", 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1mvsphARAAlhGdDUn101JhENibKBJymQ==", 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.973 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mds.d", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1mF0tODhAACXE4CInpUuYPOaRVia6HsQ==", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.0", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBHRB1m5DZ6NBAAHq3NyjnLkmy2+12CH+2wAA==", 2024-04-15T15:16:19.974 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.1", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBKRB1mOfrpMhAA1LErtVE7PABH7mYCQH5eaQ==", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.975 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.2", 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBNRB1mLOY6MRAACJAUnSukk7OtsJyIzyelxQ==", 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.976 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.977 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.3", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBQRB1m4RJ4LxAAgaJwWgQO58fok1arrpOUfA==", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.4", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBWRB1mwgY7OBAAY8KIzT+psWeTYb2Gex0f5g==", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.978 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.5", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBZRB1mzlofNxAAyFht43aiV+MwY9s81VTYMQ==", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.6", 2024-04-15T15:16:19.979 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBcRB1m+2sKNhAADnej7b5JltPOhDlqH30lgw==", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "osd.7", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBfRB1m+ZzENBAAMhNOvxpkifAOpKvnQkO/zQ==", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow profile osd", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile osd", 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.0", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1mV0TQEhAAVAq/vgY21Iz2b2lmIPnQPQ==", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.1", 2024-04-15T15:16:19.981 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1mUYdtHBAACC0J3a7+fz6P3ESJGyw1Ug==", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.2", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1ml+7MGBAAM+5SF+JwojvhE8UdWXFLLw==", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-15T15:16:19.982 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.3", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBGRB1mTpMUIBAA5Pe00R2+efb1TT5Q1vkT2g==", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow r", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow rw", 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow rwx" 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.983 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.admin", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBFRB1mTiUVGBAAnIEYxvLuISeN22OBchFUSA==", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "mgr": "allow *", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow *", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-mds", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1mPZVoMRAAHhQ5s/wQG1lR4a8yYmyVlQ==", 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-mds" 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-mgr", 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1mAp5oMRAAe8wh8blvS2vhNYbAfGoyrw==", 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-osd", 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1muaRoMRAAJv4JB24/Siez7PXo5JYUHg==", 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-osd" 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rbd", 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1mF6toMRAApFmoCKKsgWw8ea0UZ3bl+g==", 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1mmrFoMRAAYT/TMHb+94aAMb4tO1zN1w==", 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "client.bootstrap-rgw", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBlRB1mWbhoMRAA9xJw8QpwFRgQhs0b7KqWQw==", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.x", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBFRB1mGCvQMRAAx+UFS92U9dNKJ59UmwU3zQ==", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": "mgr.y", 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "key": "AQBFRB1m/2LdLRAASp0e6ytz/SwoCwpsA9IsMw==", 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "caps": { 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "mds": "allow *", 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "mon": "allow profile mgr", 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: "osd": "allow *" 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-04-15T15:16:19.988 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 msgr2=0x7f3d4806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 0x7f3d4806edb0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c15e1e0 tx=0x7f3d500079b0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 msgr2=0x7f3d5c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c0097a0 tx=0x7f3d4c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 shutdown_connections 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3d5c14f420 0x7f3d5c15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3d5c15e830 0x7f3d5c0c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3d4806c8f0 0x7f3d4806edb0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 --2- 172.21.15.70:0/2939635048 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3d5c0b53c0 0x7f3d5c15dca0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:19.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.970+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 >> 172.21.15.70:0/2939635048 conn(0x7f3d5c073360 msgr2=0x7f3d5c0af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.971+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 shutdown_connections 2024-04-15T15:16:19.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:19.971+0000 7f3d62642640 1 -- 172.21.15.70:0/2939635048 wait complete. 2024-04-15T15:16:19.991 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-15T15:16:20.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.093+0000 7f1b5eb5f640 1 Processor -- start 2024-04-15T15:16:20.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.093+0000 7f1b5eb5f640 1 -- start start 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5eb5f640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b5815e830 0x7f1b580b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5eb5f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815ec10 0x7f1b58067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5eb5f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b580b3c10 con 0x7f1b58067c60 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5eb5f640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1b580b3ef0 con 0x7f1b5815e830 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5eb5f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1b580b41d0 con 0x7f1b5815ec10 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b4ffff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815ec10 0x7f1b58067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5c8fb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1b5815e830 0x7f1b580b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5d0fc640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f1b58067c60 0x7f1b580b53c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:36528/0 (socket says 172.21.15.70:36528) 2024-04-15T15:16:20.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5d0fc640 1 -- 172.21.15.70:0/2656929073 learned_addr learned my addr 172.21.15.70:0/2656929073 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:20.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5c8fb640 1 -- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815ec10 msgr2=0x7f1b58067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5c8fb640 1 --2- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815ec10 0x7f1b58067720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.094+0000 7f1b5c8fb640 1 -- 172.21.15.70:0/2656929073 >> v1:172.21.15.96:6789/0 conn(0x7f1b58067c60 legacy=0x7f1b580b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.095+0000 7f1b4f7fe640 1 -- 172.21.15.70:0/2656929073 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382223727 0 0) 0x7f1b580b3c10 con 0x7f1b58067c60 2024-04-15T15:16:20.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.095+0000 7f1b5c8fb640 1 -- 172.21.15.70:0/2656929073 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b580b44b0 con 0x7f1b5815e830 2024-04-15T15:16:20.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.095+0000 7f1b5c8fb640 1 --2- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b5815e830 0x7f1b580b4cb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d90 tx=0x7f1b5402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=754146a54c06dcdc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.095+0000 7f1b4f7fe640 1 -- 172.21.15.70:0/2656929073 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54039070 con 0x7f1b5815e830 2024-04-15T15:16:20.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b4f7fe640 1 -- 172.21.15.70:0/2656929073 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b5402cc50 con 0x7f1b5815e830 2024-04-15T15:16:20.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b4f7fe640 1 -- 172.21.15.70:0/2656929073 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54034440 con 0x7f1b5815e830 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b5815e830 msgr2=0x7f1b580b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b5815e830 0x7f1b580b4cb0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1b54009d90 tx=0x7f1b5402bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/2656929073 shutdown_connections 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815ec10 0x7f1b58067720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/2656929073 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b5815e830 0x7f1b580b4cb0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/2656929073 >> 172.21.15.70:0/2656929073 conn(0x7f1b58073360 msgr2=0x7f1b5806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/2656929073 shutdown_connections 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.096+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/2656929073 wait complete. 2024-04-15T15:16:20.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.097+0000 7f1b5eb5f640 1 Processor -- start 2024-04-15T15:16:20.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.097+0000 7f1b5eb5f640 1 -- start start 2024-04-15T15:16:20.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b58067c60 0x7f1b5815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815e830 0x7f1b581456f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1b581600e0 con 0x7f1b58155ca0 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1b5815ff60 con 0x7f1b58067c60 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5eb5f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1b580b5ad0 con 0x7f1b5815e830 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5d0fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5d0fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45024/0 (socket says 172.21.15.70:45024) 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.098+0000 7f1b5d0fc640 1 -- 172.21.15.70:0/3741572179 learned_addr learned my addr 172.21.15.70:0/3741572179 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b4ffff640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815e830 0x7f1b581456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5c8fb640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b58067c60 0x7f1b5815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 -- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815e830 msgr2=0x7f1b581456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815e830 0x7f1b581456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 -- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b58067c60 msgr2=0x7f1b5815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b58067c60 0x7f1b5815b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 -- 172.21.15.70:0/3741572179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b58156210 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5d0fc640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c480 tx=0x7f1b5000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50018070 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b50007500 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b581564f0 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b50004a20 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.099+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b58146420 con 0x7f1b58155ca0 2024-04-15T15:16:20.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.100+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b580689e0 con 0x7f1b58155ca0 2024-04-15T15:16:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.101+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1b5001b070 con 0x7f1b58155ca0 2024-04-15T15:16:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.101+0000 7f1b4d7fa640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 0x7f1b4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.101+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f1b5008ab50 con 0x7f1b58155ca0 2024-04-15T15:16:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.101+0000 7f1b5c8fb640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 0x7f1b4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.102+0000 7f1b5c8fb640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 0x7f1b4406edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b54036000 tx=0x7f1b54030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.106+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1b50058b80 con 0x7f1b58155ca0 2024-04-15T15:16:20.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.348+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f1b5806fc40 con 0x7f1b58155ca0 2024-04-15T15:16:20.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.350+0000 7f1b4d7fa640 1 -- 172.21.15.70:0/3741572179 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f1b5005ce80 con 0x7f1b58155ca0 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 msgr2=0x7f1b4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 0x7f1b4406edb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b54036000 tx=0x7f1b54030000 comp rx=0 tx=0).stop 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 msgr2=0x7f1b580c5ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c480 tx=0x7f1b5000c940 comp rx=0 tx=0).stop 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 shutdown_connections 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1b4406c8f0 0x7f1b4406edb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1b58155ca0 0x7f1b580c5ae0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1b5815e830 0x7f1b581456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 --2- 172.21.15.70:0/3741572179 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1b58067c60 0x7f1b5815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 >> 172.21.15.70:0/3741572179 conn(0x7f1b58073360 msgr2=0x7f1b580af7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 shutdown_connections 2024-04-15T15:16:20.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.352+0000 7f1b5eb5f640 1 -- 172.21.15.70:0/3741572179 wait complete. 2024-04-15T15:16:20.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-15T15:16:20.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.467+0000 7fe54a0d3640 1 Processor -- start 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.467+0000 7fe54a0d3640 1 -- start start 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe54a0d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe544158770 0x7fe54415bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe54a0d3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5440b5230 0x7fe544072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe54a0d3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe54415c1d0 con 0x7fe544072c60 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe54a0d3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe5440b4da0 con 0x7fe5440b5230 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe54a0d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe5440b4190 con 0x7fe544158770 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe542ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5440b5230 0x7fe544072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe543fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe544072c60 0x7fe54415e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54628/0 (socket says 172.21.15.70:54628) 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe543fff640 1 -- 172.21.15.70:0/1722591913 learned_addr learned my addr 172.21.15.70:0/1722591913 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe542ffd640 1 -- 172.21.15.70:0/1722591913 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe544158770 msgr2=0x7fe54415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe542ffd640 1 --2- 172.21.15.70:0/1722591913 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe544158770 0x7fe54415bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe542ffd640 1 -- 172.21.15.70:0/1722591913 >> v1:172.21.15.70:6789/0 conn(0x7fe544072c60 legacy=0x7fe54415e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe542ffd640 1 -- 172.21.15.70:0/1722591913 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5440b4470 con 0x7fe5440b5230 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.468+0000 7fe5427fc640 1 -- 172.21.15.70:0/1722591913 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1280404670 0 0) 0x7fe53c001490 con 0x7fe544072c60 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe542ffd640 1 --2- 172.21.15.70:0/1722591913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5440b5230 0x7fe544072720 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe5380097a0 tx=0x7fe53802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95689aaf3766ea08 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe5427fc640 1 -- 172.21.15.70:0/1722591913 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538035020 con 0x7fe5440b5230 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe5427fc640 1 -- 172.21.15.70:0/1722591913 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe538031830 con 0x7fe5440b5230 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe5427fc640 1 -- 172.21.15.70:0/1722591913 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe538031c00 con 0x7fe5440b5230 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 -- 172.21.15.70:0/1722591913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5440b5230 msgr2=0x7fe544072720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/1722591913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5440b5230 0x7fe544072720 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fe5380097a0 tx=0x7fe53802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 -- 172.21.15.70:0/1722591913 shutdown_connections 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/1722591913 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5440b5230 0x7fe544072720 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/1722591913 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe544158770 0x7fe54415bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 -- 172.21.15.70:0/1722591913 >> 172.21.15.70:0/1722591913 conn(0x7fe54406be00 msgr2=0x7fe54406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 -- 172.21.15.70:0/1722591913 shutdown_connections 2024-04-15T15:16:20.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.469+0000 7fe54a0d3640 1 -- 172.21.15.70:0/1722591913 wait complete. 2024-04-15T15:16:20.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.471+0000 7fe54a0d3640 1 Processor -- start 2024-04-15T15:16:20.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.471+0000 7fe54a0d3640 1 -- start start 2024-04-15T15:16:20.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5440b5230 0x7fe54414bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe544158770 0x7fe5440cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5440b4a30 con 0x7fe544072c60 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5440730b0 con 0x7fe544158770 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe544071fa0 con 0x7fe5440b5230 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45038/0 (socket says 172.21.15.70:45038) 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 -- 172.21.15.70:0/4078697898 learned_addr learned my addr 172.21.15.70:0/4078697898 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 -- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5440b5230 msgr2=0x7fe54414bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5440b5230 0x7fe54414bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 -- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe544158770 msgr2=0x7fe5440cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe544158770 0x7fe5440cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 -- 172.21.15.70:0/4078697898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe54414c500 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe5437fe640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe53400b410 tx=0x7fe53400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534015020 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe534011850 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5441597b0 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe534011c40 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.472+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe544159d70 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.473+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5440689e0 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.474+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fe534018020 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.474+0000 7fe52bfff640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 0x7fe53006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.474+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fe53408b870 con 0x7fe544072c60 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.474+0000 7fe542ffd640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 0x7fe53006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.475+0000 7fe542ffd640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 0x7fe53006edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.478+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe53401f020 con 0x7fe544072c60 2024-04-15T15:16:20.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.724+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7fe5440b3790 con 0x7fe544072c60 2024-04-15T15:16:20.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.726+0000 7fe52bfff640 1 -- 172.21.15.70:0/4078697898 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7fe5340172c0 con 0x7fe544072c60 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 msgr2=0x7fe53006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 0x7fe53006edb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe538036000 tx=0x7fe538030000 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 msgr2=0x7fe5440af0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fe53400b410 tx=0x7fe53400b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 shutdown_connections 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe544158770 0x7fe5440cc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5440b5230 0x7fe54414bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe53006c8f0 0x7fe53006edb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 --2- 172.21.15.70:0/4078697898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe544072c60 0x7fe5440af0b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 >> 172.21.15.70:0/4078697898 conn(0x7fe54406be00 msgr2=0x7fe54415e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.730 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 shutdown_connections 2024-04-15T15:16:20.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.728+0000 7fe54a0d3640 1 -- 172.21.15.70:0/4078697898 wait complete. 2024-04-15T15:16:20.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.843+0000 7f04c4c88640 1 Processor -- start 2024-04-15T15:16:20.845 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.843+0000 7f04c4c88640 1 -- start start 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04c4c88640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04c00d2360 0x7f04c00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04c4c88640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c00b47a0 con 0x7f04c015e700 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04c4c88640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04c00b4a80 con 0x7f04c015eae0 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04c4c88640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f04c00b53b0 con 0x7f04c00d2360 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04b7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04c00d2360 0x7f04c00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04be59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f04c015e700 0x7f04c00b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54634/0 (socket says 172.21.15.70:54634) 2024-04-15T15:16:20.846 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04be59c640 1 -- 172.21.15.70:0/4275208427 learned_addr learned my addr 172.21.15.70:0/4275208427 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04bdd9b640 1 -- 172.21.15.70:0/4275208427 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644132643 0 0) 0x7f04c00b47a0 con 0x7f04c015e700 2024-04-15T15:16:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04bdd9b640 1 -- 172.21.15.70:0/4275208427 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04ac003520 con 0x7f04c015e700 2024-04-15T15:16:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04b7fff640 1 -- 172.21.15.70:0/4275208427 >> v1:172.21.15.96:6789/0 conn(0x7f04c015eae0 legacy=0x7f04c00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04b7fff640 1 -- 172.21.15.70:0/4275208427 >> v1:172.21.15.70:6789/0 conn(0x7f04c015e700 legacy=0x7f04c00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.847 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.844+0000 7f04b7fff640 1 -- 172.21.15.70:0/4275208427 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c00b4d60 con 0x7f04c00d2360 2024-04-15T15:16:20.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04b7fff640 1 --2- 172.21.15.70:0/4275208427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c00d2360 0x7f04c00d2b60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f04c00b5530 tx=0x7f04a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291b24e2c5b93e6f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04bdd9b640 1 -- 172.21.15.70:0/4275208427 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8035020 con 0x7f04c00d2360 2024-04-15T15:16:20.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04bdd9b640 1 -- 172.21.15.70:0/4275208427 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04a8031830 con 0x7f04c00d2360 2024-04-15T15:16:20.848 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04bdd9b640 1 -- 172.21.15.70:0/4275208427 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04a8031c00 con 0x7f04c00d2360 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04c4c88640 1 -- 172.21.15.70:0/4275208427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c00d2360 msgr2=0x7f04c00d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.845+0000 7f04c4c88640 1 --2- 172.21.15.70:0/4275208427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c00d2360 0x7f04c00d2b60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f04c00b5530 tx=0x7f04a802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.846+0000 7f04c4c88640 1 -- 172.21.15.70:0/4275208427 shutdown_connections 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.846+0000 7f04c4c88640 1 --2- 172.21.15.70:0/4275208427 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c00d2360 0x7f04c00d2b60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.846+0000 7f04c4c88640 1 -- 172.21.15.70:0/4275208427 >> 172.21.15.70:0/4275208427 conn(0x7f04c008bf60 msgr2=0x7f04c00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.846+0000 7f04c4c88640 1 -- 172.21.15.70:0/4275208427 shutdown_connections 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.846+0000 7f04c4c88640 1 -- 172.21.15.70:0/4275208427 wait complete. 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.847+0000 7f04c4c88640 1 Processor -- start 2024-04-15T15:16:20.849 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.847+0000 7f04c4c88640 1 -- start start 2024-04-15T15:16:20.852 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c00d2360 0x7f04c01618e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 0x7f04c01527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04c00d1d40 con 0x7f04c015eae0 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04c00b4450 con 0x7f04c015e700 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f04c00d1ec0 con 0x7f04c00d2360 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45050/0 (socket says 172.21.15.70:45050) 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 -- 172.21.15.70:0/1385167625 learned_addr learned my addr 172.21.15.70:0/1385167625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04be59c640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c00d2360 0x7f04c01618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04b7fff640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 0x7f04c01527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 -- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c00d2360 msgr2=0x7f04c01618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c00d2360 0x7f04c01618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 -- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 msgr2=0x7f04c01527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:20.854 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 0x7f04c01527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 -- 172.21.15.70:0/1385167625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c0160560 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04b7fff640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 0x7f04c01527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04bed9d640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f04b800b9a0 tx=0x7f04b800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8019070 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b800cba0 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c0160840 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8014440 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.848+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c0160e00 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.849+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c00689e0 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.850+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f04b801c070 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.850+0000 7f04b6ffd640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 0x7f04a006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.850+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f04b808b5a0 con 0x7f04c015eae0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.850+0000 7f04be59c640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 0x7f04a006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:20.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.851+0000 7f04be59c640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 0x7f04a006ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f04b0001360 tx=0x7f04b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:20.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:20.855+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04b8059b90 con 0x7f04c015eae0 2024-04-15T15:16:21.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.096+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f04c00ba500 con 0x7f04c015eae0 2024-04-15T15:16:21.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.098+0000 7f04b6ffd640 1 -- 172.21.15.70:0/1385167625 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f04b8059490 con 0x7f04c015eae0 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 msgr2=0x7f04a006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 0x7f04a006ede0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f04b0001360 tx=0x7f04b0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 msgr2=0x7f04c015c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f04b800b9a0 tx=0x7f04b800be60 comp rx=0 tx=0).stop 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 shutdown_connections 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a006c920 0x7f04a006ede0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c015eae0 0x7f04c015c4f0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c015e700 0x7f04c01527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 --2- 172.21.15.70:0/1385167625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c00d2360 0x7f04c01618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 >> 172.21.15.70:0/1385167625 conn(0x7f04c008bf60 msgr2=0x7f04c006e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 shutdown_connections 2024-04-15T15:16:21.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.100+0000 7f04c4c88640 1 -- 172.21.15.70:0/1385167625 wait complete. 2024-04-15T15:16:21.113 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.216+0000 7f87477e2640 1 Processor -- start 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.216+0000 7f87477e2640 1 -- start start 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f87477e2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 0x7f87400b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f87477e2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87400b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f87477e2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8740067e30 con 0x7f874015ec10 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f87477e2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f87400b3c10 con 0x7f874015e830 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f87477e2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f87400b3ef0 con 0x7f87400b53c0 2024-04-15T15:16:21.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8744d7d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f874015ec10 0x7f8740067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54640/0 (socket says 172.21.15.70:54640) 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8744d7d640 1 -- 172.21.15.70:0/304047679 learned_addr learned my addr 172.21.15.70:0/304047679 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f874557e640 1 --2- 172.21.15.70:0/304047679 >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 0x7f87400b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8745d7f640 1 --2- 172.21.15.70:0/304047679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87400b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4183754501 0 0) 0x7f8740067e30 con 0x7f874015ec10 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f872c003520 con 0x7f874015ec10 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2234485757 0 0) 0x7f872c003520 con 0x7f874015ec10 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8745d7f640 1 -- 172.21.15.70:0/304047679 >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 msgr2=0x7f87400b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8745d7f640 1 --2- 172.21.15.70:0/304047679 >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 0x7f87400b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8745d7f640 1 -- 172.21.15.70:0/304047679 >> v1:172.21.15.70:6789/0 conn(0x7f874015ec10 legacy=0x7f8740067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f8745d7f640 1 -- 172.21.15.70:0/304047679 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87400b41d0 con 0x7f87400b53c0 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.217+0000 7f874557e640 1 --2- 172.21.15.70:0/304047679 >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 0x7f87400b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:21.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.218+0000 7f8745d7f640 1 --2- 172.21.15.70:0/304047679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87400b5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8738009d90 tx=0x7f873802bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d12864c02156e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.218+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738039070 con 0x7f87400b53c0 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.218+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f873802c8a0 con 0x7f87400b53c0 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.218+0000 7f8737fff640 1 -- 172.21.15.70:0/304047679 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8730001670 con 0x7f87400b53c0 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 -- 172.21.15.70:0/304047679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 msgr2=0x7f87400b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 --2- 172.21.15.70:0/304047679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87400b5840 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8738009d90 tx=0x7f873802bd70 comp rx=0 tx=0).stop 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 -- 172.21.15.70:0/304047679 shutdown_connections 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 --2- 172.21.15.70:0/304047679 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87400b5840 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 --2- 172.21.15.70:0/304047679 >> v2:172.21.15.96:3300/0 conn(0x7f874015e830 0x7f87400b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 -- 172.21.15.70:0/304047679 >> 172.21.15.70:0/304047679 conn(0x7f8740073360 msgr2=0x7f874006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 -- 172.21.15.70:0/304047679 shutdown_connections 2024-04-15T15:16:21.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.219+0000 7f87477e2640 1 -- 172.21.15.70:0/304047679 wait complete. 2024-04-15T15:16:21.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.220+0000 7f87477e2640 1 Processor -- start 2024-04-15T15:16:21.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.220+0000 7f87477e2640 1 -- start start 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87401f4b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 0x7f87401f79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f874015ff60 con 0x7f8740154810 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f87400b4790 con 0x7f874015e830 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f87477e2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f87401600e0 con 0x7f87400b53c0 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45058/0 (socket says 172.21.15.70:45058) 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 -- 172.21.15.70:0/1450454324 learned_addr learned my addr 172.21.15.70:0/1450454324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f874557e640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87401f4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8744d7d640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 0x7f87401f79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 -- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 msgr2=0x7f87401f4b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87401f4b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 -- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 msgr2=0x7f87401f79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 0x7f87401f79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8745d7f640 1 -- 172.21.15.70:0/1450454324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8740155500 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.221+0000 7f8744d7d640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 0x7f87401f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f8745d7f640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8738009eb0 tx=0x7f8738004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738039050 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8738045070 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87401557e0 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8738009450 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.222+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8740155da0 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.223+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87400b5f60 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.224+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8738040690 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.224+0000 7f8735ffb640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 0x7f872406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.224+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f87380ae1a0 con 0x7f8740154810 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.224+0000 7f874557e640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 0x7f872406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.224+0000 7f874557e640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 0x7f872406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f873c009560 tx=0x7f873c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.228+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f87400b5f60 con 0x7f8740154810 2024-04-15T15:16:21.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.472+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f87400b6540 con 0x7f8740154810 2024-04-15T15:16:21.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.474+0000 7f8735ffb640 1 -- 172.21.15.70:0/1450454324 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f87400b6540 con 0x7f8740154810 2024-04-15T15:16:21.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 msgr2=0x7f872406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 0x7f872406edb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f873c009560 tx=0x7f873c009190 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 msgr2=0x7f8740154cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f8738009eb0 tx=0x7f8738004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 shutdown_connections 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f872406c8f0 0x7f872406edb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8740154810 0x7f8740154cc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f874015e830 0x7f87401f79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 --2- 172.21.15.70:0/1450454324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f87400b53c0 0x7f87401f4b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.476+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 >> 172.21.15.70:0/1450454324 conn(0x7f8740073360 msgr2=0x7f874006e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.477+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 shutdown_connections 2024-04-15T15:16:21.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.477+0000 7f87477e2640 1 -- 172.21.15.70:0/1450454324 wait complete. 2024-04-15T15:16:21.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-15T15:16:21.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.601+0000 7fcddf172640 1 Processor -- start 2024-04-15T15:16:21.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.601+0000 7fcddf172640 1 -- start start 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcdd8072d80 0x7fcdd8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdd80af3e0 0x7fcdd80af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd80d0690 0x7fcdd80d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd815db40 con 0x7fcdd80af3e0 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcdd815cfc0 con 0x7fcdd8072d80 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddf172640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd815d2a0 con 0x7fcdd80d0690 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdd80af3e0 0x7fcdd80af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcdd80af3e0 0x7fcdd80af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45066/0 (socket says 172.21.15.70:45066) 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 -- 172.21.15.70:0/3105048743 learned_addr learned my addr 172.21.15.70:0/3105048743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddcf0e640 1 --2- 172.21.15.70:0/3105048743 >> v2:172.21.15.96:3300/0 conn(0x7fcdd8072d80 0x7fcdd8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.605 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcddd70f640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd80d0690 0x7fcdd80d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 -- 172.21.15.70:0/3105048743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd80d0690 msgr2=0x7fcdd80d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd80d0690 0x7fcdd80d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 -- 172.21.15.70:0/3105048743 >> v2:172.21.15.96:3300/0 conn(0x7fcdd8072d80 msgr2=0x7fcdd8071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/3105048743 >> v2:172.21.15.96:3300/0 conn(0x7fcdd8072d80 0x7fcdd8071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.602+0000 7fcdd7fff640 1 -- 172.21.15.70:0/3105048743 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd815d580 con 0x7fcdd80af3e0 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.603+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd80af840 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80d11c0 tx=0x7fcdc402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b698e6c309e1d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.603+0000 7fcdd77fe640 1 -- 172.21.15.70:0/3105048743 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc402e070 con 0x7fcdd80af3e0 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.603+0000 7fcdd77fe640 1 -- 172.21.15.70:0/3105048743 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc4037070 con 0x7fcdd80af3e0 2024-04-15T15:16:21.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.603+0000 7fcdd77fe640 1 -- 172.21.15.70:0/3105048743 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4031a00 con 0x7fcdd80af3e0 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 -- 172.21.15.70:0/3105048743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 msgr2=0x7fcdd80af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd80af840 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcdd80d11c0 tx=0x7fcdc402b880 comp rx=0 tx=0).stop 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 -- 172.21.15.70:0/3105048743 shutdown_connections 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd80d0690 0x7fcdd80d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 --2- 172.21.15.70:0/3105048743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd80af840 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 --2- 172.21.15.70:0/3105048743 >> v2:172.21.15.96:3300/0 conn(0x7fcdd8072d80 0x7fcdd8071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 -- 172.21.15.70:0/3105048743 >> 172.21.15.70:0/3105048743 conn(0x7fcdd806be00 msgr2=0x7fcdd806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 -- 172.21.15.70:0/3105048743 shutdown_connections 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.604+0000 7fcddf172640 1 -- 172.21.15.70:0/3105048743 wait complete. 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.605+0000 7fcddf172640 1 Processor -- start 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.605+0000 7fcddf172640 1 -- start start 2024-04-15T15:16:21.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd8072d80 0x7fcdd8148600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdd80d0690 0x7fcdd815e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd815cd10 con 0x7fcdd80af3e0 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd815cb90 con 0x7fcdd80d0690 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddf172640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcdd806e060 con 0x7fcdd8072d80 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45068/0 (socket says 172.21.15.70:45068) 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 -- 172.21.15.70:0/1805937865 learned_addr learned my addr 172.21.15.70:0/1805937865 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddcf0e640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd8072d80 0x7fcdd8148600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcddd70f640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdd80d0690 0x7fcdd815e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 -- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd8072d80 msgr2=0x7fcdd8148600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd8072d80 0x7fcdd8148600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 -- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdd80d0690 msgr2=0x7fcdd815e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdd80d0690 0x7fcdd815e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 -- 172.21.15.70:0/1805937865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd8152d00 con 0x7fcdd80af3e0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.606+0000 7fcdd7fff640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4000c00 tx=0x7fcdc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.607+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4032670 con 0x7fcdd80af3e0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.607+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdc4032850 con 0x7fcdd80af3e0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.607+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd80b38b0 con 0x7fcdd80af3e0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.607+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4032c40 con 0x7fcdd80af3e0 2024-04-15T15:16:21.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.607+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd8157250 con 0x7fcdd80af3e0 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.608+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd80689e0 con 0x7fcdd80af3e0 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.608+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcdc404c060 con 0x7fcdd80af3e0 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.609+0000 7fcdd57fa640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 0x7fcdbc06ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.609+0000 7fcddcf0e640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 0x7fcdbc06ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.609+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fcdc40ad330 con 0x7fcdd80af3e0 2024-04-15T15:16:21.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.609+0000 7fcddcf0e640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 0x7fcdbc06ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.613+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcdd8152d00 con 0x7fcdd80af3e0 2024-04-15T15:16:21.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.858+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fcdd8156b70 con 0x7fcdd80af3e0 2024-04-15T15:16:21.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.860+0000 7fcdd57fa640 1 -- 172.21.15.70:0/1805937865 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fcdc4040070 con 0x7fcdd80af3e0 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 msgr2=0x7fcdbc06ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 0x7fcdbc06ede0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 msgr2=0x7fcdd8157d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4000c00 tx=0x7fcdc40046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 shutdown_connections 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcdd80d0690 0x7fcdd815e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcdbc06c920 0x7fcdbc06ede0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcdd80af3e0 0x7fcdd8157d30 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 --2- 172.21.15.70:0/1805937865 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcdd8072d80 0x7fcdd8148600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 >> 172.21.15.70:0/1805937865 conn(0x7fcdd806be00 msgr2=0x7fcdd80d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 shutdown_connections 2024-04-15T15:16:21.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.862+0000 7fcddf172640 1 -- 172.21.15.70:0/1805937865 wait complete. 2024-04-15T15:16:21.875 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-15T15:16:21.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.978+0000 7f8e3bfba640 1 Processor -- start 2024-04-15T15:16:21.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.978+0000 7f8e3bfba640 1 -- start start 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3bfba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8e3406f6d0 0x7f8e340b3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3bfba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e340d0570 0x7f8e340d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3bfba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e3406fab0 con 0x7f8e3415f5c0 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3bfba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8e340b49a0 con 0x7f8e340d0570 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3bfba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8e340b43e0 con 0x7f8e3406f6d0 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e340d0570 0x7f8e340d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8e340d0570 0x7f8e340d0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45072/0 (socket says 172.21.15.70:45072) 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 -- 172.21.15.70:0/1016523265 learned_addr learned my addr 172.21.15.70:0/1016523265 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:21.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e39d56640 1 --2- 172.21.15.70:0/1016523265 >> v2:172.21.15.96:3300/0 conn(0x7f8e3406f6d0 0x7f8e340b3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 -- 172.21.15.70:0/1016523265 >> v2:172.21.15.96:3300/0 conn(0x7f8e3406f6d0 msgr2=0x7f8e340b3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 --2- 172.21.15.70:0/1016523265 >> v2:172.21.15.96:3300/0 conn(0x7f8e3406f6d0 0x7f8e340b3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 -- 172.21.15.70:0/1016523265 >> v1:172.21.15.96:6789/0 conn(0x7f8e3415f5c0 legacy=0x7f8e340729b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:21.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.979+0000 7f8e3a557640 1 -- 172.21.15.70:0/1016523265 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e340b46c0 con 0x7f8e340d0570 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e3a557640 1 --2- 172.21.15.70:0/1016523265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e340d0570 0x7f8e340d0a00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e3406da10 tx=0x7f8e2c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407ec3bb82cecf74 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e38d54640 1 -- 172.21.15.70:0/1016523265 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c02cb20 con 0x7f8e340d0570 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e38d54640 1 -- 172.21.15.70:0/1016523265 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2c02ccc0 con 0x7f8e340d0570 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e38d54640 1 -- 172.21.15.70:0/1016523265 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c034610 con 0x7f8e340d0570 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1016523265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e340d0570 msgr2=0x7f8e340d0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.980+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1016523265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e340d0570 0x7f8e340d0a00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8e3406da10 tx=0x7f8e2c02bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:21.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1016523265 shutdown_connections 2024-04-15T15:16:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1016523265 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e340d0570 0x7f8e340d0a00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1016523265 >> v2:172.21.15.96:3300/0 conn(0x7f8e3406f6d0 0x7f8e340b3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1016523265 >> 172.21.15.70:0/1016523265 conn(0x7f8e3406b9e0 msgr2=0x7f8e3406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1016523265 shutdown_connections 2024-04-15T15:16:21.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.981+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1016523265 wait complete. 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.982+0000 7f8e3bfba640 1 Processor -- start 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.982+0000 7f8e3bfba640 1 -- start start 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 0x7f8e34140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8e340d0570 0x7f8e34143440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3415fa70 con 0x7f8e3415f5c0 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8e34072490 con 0x7f8e3406f6d0 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8e3406e5b0 con 0x7f8e340d0570 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45084/0 (socket says 172.21.15.70:45084) 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 -- 172.21.15.70:0/1412746825 learned_addr learned my addr 172.21.15.70:0/1412746825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e39d56640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 0x7f8e34140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e39555640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8e340d0570 0x7f8e34143440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 -- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8e340d0570 msgr2=0x7f8e34143440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8e340d0570 0x7f8e34143440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 -- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 msgr2=0x7f8e34140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 0x7f8e34140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 -- 172.21.15.70:0/1412746825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e340ceef0 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e39d56640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 0x7f8e34140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3a557640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009f40 tx=0x7f8e2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c043020 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2c03e450 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e340ce370 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e2c009510 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.983+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e340ce930 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.984+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e340689e0 con 0x7f8e3415f5c0 2024-04-15T15:16:21.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.985+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8e2c005d10 con 0x7f8e3415f5c0 2024-04-15T15:16:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.986+0000 7f8e267fc640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 0x7f8e1806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.986+0000 7f8e39d56640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 0x7f8e1806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.986+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f8e2c0ae3a0 con 0x7f8e3415f5c0 2024-04-15T15:16:21.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.986+0000 7f8e39d56640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 0x7f8e1806ede0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e28006fd0 tx=0x7f8e28008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:21.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:21.990+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8e2c07ca20 con 0x7f8e3415f5c0 2024-04-15T15:16:22.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.237+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f8e34141480 con 0x7f8e3415f5c0 2024-04-15T15:16:22.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.239+0000 7f8e267fc640 1 -- 172.21.15.70:0/1412746825 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f8e2c07c320 con 0x7f8e3415f5c0 2024-04-15T15:16:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 msgr2=0x7f8e1806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 0x7f8e1806ede0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8e28006fd0 tx=0x7f8e28008040 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 msgr2=0x7f8e34143980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8e2c009f40 tx=0x7f8e2c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 shutdown_connections 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8e1806c920 0x7f8e1806ede0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e3415f5c0 0x7f8e34143980 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8e340d0570 0x7f8e34143440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 --2- 172.21.15.70:0/1412746825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8e3406f6d0 0x7f8e34140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 >> 172.21.15.70:0/1412746825 conn(0x7f8e3406b9e0 msgr2=0x7f8e340b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 shutdown_connections 2024-04-15T15:16:22.244 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.241+0000 7f8e3bfba640 1 -- 172.21.15.70:0/1412746825 wait complete. 2024-04-15T15:16:22.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:22.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.355+0000 7f8f23d1c640 1 Processor -- start 2024-04-15T15:16:22.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.356+0000 7f8f23d1c640 1 -- start start 2024-04-15T15:16:22.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.356+0000 7f8f23d1c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8f1c15e700 0x7f8f1c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.356+0000 7f8f23d1c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8f1c0d2360 0x7f8f1c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f23d1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f1c0b47a0 con 0x7f8f1c15eae0 2024-04-15T15:16:22.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f23d1c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1c0b53b0 con 0x7f8f1c15e700 2024-04-15T15:16:22.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f23d1c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8f1c0b4a80 con 0x7f8f1c0d2360 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f21ab8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8f1c15e700 0x7f8f1c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f222b9640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8f1c15eae0 0x7f8f1c0cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:54650/0 (socket says 172.21.15.70:54650) 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f222b9640 1 -- 172.21.15.70:0/2955162632 learned_addr learned my addr 172.21.15.70:0/2955162632 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2955162632 >> v2:172.21.15.96:3300/0 conn(0x7f8f1c0d2360 msgr2=0x7f8f1c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f21ab8640 1 --2- 172.21.15.70:0/2955162632 >> v2:172.21.15.96:3300/0 conn(0x7f8f1c0d2360 0x7f8f1c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2955162632 >> v1:172.21.15.70:6789/0 conn(0x7f8f1c15eae0 legacy=0x7f8f1c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f20ab6640 1 -- 172.21.15.70:0/2955162632 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3799220671 0 0) 0x7f8f1c0b47a0 con 0x7f8f1c15eae0 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.357+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2955162632 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c0b4d60 con 0x7f8f1c15e700 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f21ab8640 1 --2- 172.21.15.70:0/2955162632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c15e700 0x7f8f1c0b5840 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c08c7b0 tx=0x7f8f1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e07e444183ed74 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f20ab6640 1 -- 172.21.15.70:0/2955162632 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10035020 con 0x7f8f1c15e700 2024-04-15T15:16:22.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f20ab6640 1 -- 172.21.15.70:0/2955162632 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10031830 con 0x7f8f1c15e700 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f20ab6640 1 -- 172.21.15.70:0/2955162632 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10031c00 con 0x7f8f1c15e700 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2955162632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c15e700 msgr2=0x7f8f1c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2955162632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c15e700 0x7f8f1c0b5840 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c08c7b0 tx=0x7f8f1002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2955162632 shutdown_connections 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2955162632 >> v2:172.21.15.96:3300/0 conn(0x7f8f1c0d2360 0x7f8f1c0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2955162632 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c15e700 0x7f8f1c0b5840 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.358+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2955162632 >> 172.21.15.70:0/2955162632 conn(0x7f8f1c08bf60 msgr2=0x7f8f1c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.359+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2955162632 shutdown_connections 2024-04-15T15:16:22.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.359+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2955162632 wait complete. 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 Processor -- start 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 -- start start 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8f1c15e700 0x7f8f1c0ca990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 0x7f8f1c15bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c0b5d80 con 0x7f8f1c0d2360 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8f1c0b4450 con 0x7f8f1c15eae0 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.360+0000 7f8f23d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8f1c0d1d40 con 0x7f8f1c15e700 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45102/0 (socket says 172.21.15.70:45102) 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2533755322 learned_addr learned my addr 172.21.15.70:0/2533755322 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f222b9640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 0x7f8f1c15bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f212b7640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8f1c15e700 0x7f8f1c0ca990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8f1c15e700 msgr2=0x7f8f1c0ca990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8f1c15e700 0x7f8f1c0ca990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 msgr2=0x7f8f1c15bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 0x7f8f1c15bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 -- 172.21.15.70:0/2533755322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1c0caed0 con 0x7f8f1c0d2360 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f222b9640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 0x7f8f1c15bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f21ab8640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009880 tx=0x7f8f100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f10005e60 con 0x7f8f1c0d2360 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f10004020 con 0x7f8f1c0d2360 2024-04-15T15:16:22.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1c0b3520 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f100043f0 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.361+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f1c0b3dc0 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.362+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f1c0689e0 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.363+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8f10038020 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.363+0000 7f8f0e7fc640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 0x7f8f0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.363+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f8f100ada60 con 0x7f8f1c0d2360 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.363+0000 7f8f212b7640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 0x7f8f0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.363+0000 7f8f212b7640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 0x7f8f0006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0ba4b0 tx=0x7f8f18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.367+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8f1007c060 con 0x7f8f1c0d2360 2024-04-15T15:16:22.544 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.543+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8f1c0c8190 con 0x7f8f1c0d2360 2024-04-15T15:16:22.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.543+0000 7f8f0e7fc640 1 -- 172.21.15.70:0/2533755322 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13104 (secure 0 0 0) 0x7f8f1007b960 con 0x7f8f1c0d2360 2024-04-15T15:16:22.545 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:22.545 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":35,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:19.597052+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 msgr2=0x7f8f0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 0x7f8f0006edb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c0ba4b0 tx=0x7f8f18008040 comp rx=0 tx=0).stop 2024-04-15T15:16:22.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 msgr2=0x7f8f1c0b1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f8f10009880 tx=0x7f8f100046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 shutdown_connections 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8f1c15eae0 0x7f8f1c15bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8f1c15e700 0x7f8f1c0ca990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8f0006c8f0 0x7f8f0006edb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 --2- 172.21.15.70:0/2533755322 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8f1c0d2360 0x7f8f1c0b1160 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 >> 172.21.15.70:0/2533755322 conn(0x7f8f1c08bf60 msgr2=0x7f8f1c15ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 shutdown_connections 2024-04-15T15:16:22.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.545+0000 7f8f23d1c640 1 -- 172.21.15.70:0/2533755322 wait complete. 2024-04-15T15:16:22.559 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:22.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.664+0000 7f3162057640 1 Processor -- start 2024-04-15T15:16:22.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.664+0000 7f3162057640 1 -- start start 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f3162057640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f315c071d10 0x7f315c0d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f3162057640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315c15db40 con 0x7f315c072d80 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f3162057640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f315c072170 con 0x7f315c0d0bd0 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f3162057640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f315c0d1040 con 0x7f315c071d10 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f315c071d10 0x7f315c0d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315b7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f315c072d80 0x7f315c0af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38824/0 (socket says 172.21.15.70:38824) 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315b7fe640 1 -- 172.21.15.70:0/3776574846 learned_addr learned my addr 172.21.15.70:0/3776574846 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:22.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315affd640 1 -- 172.21.15.70:0/3776574846 >> v1:172.21.15.96:6789/0 conn(0x7f315c0d0bd0 legacy=0x7f315c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315affd640 1 -- 172.21.15.70:0/3776574846 >> v1:172.21.15.70:6789/0 conn(0x7f315c072d80 legacy=0x7f315c0af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315a7fc640 1 -- 172.21.15.70:0/3776574846 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 344536421 0 0) 0x7f315c15db40 con 0x7f315c072d80 2024-04-15T15:16:22.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.665+0000 7f315affd640 1 -- 172.21.15.70:0/3776574846 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c15cfc0 con 0x7f315c071d10 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.666+0000 7f315affd640 1 --2- 172.21.15.70:0/3776574846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c071d10 0x7f315c0d0690 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3150009d90 tx=0x7f315002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eff4760d7516d0ac server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.666+0000 7f315a7fc640 1 -- 172.21.15.70:0/3776574846 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150039070 con 0x7f315c071d10 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.666+0000 7f315a7fc640 1 -- 172.21.15.70:0/3776574846 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315002cc60 con 0x7f315c071d10 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.666+0000 7f315a7fc640 1 -- 172.21.15.70:0/3776574846 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150034660 con 0x7f315c071d10 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 -- 172.21.15.70:0/3776574846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c071d10 msgr2=0x7f315c0d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 --2- 172.21.15.70:0/3776574846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c071d10 0x7f315c0d0690 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f3150009d90 tx=0x7f315002beb0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 -- 172.21.15.70:0/3776574846 shutdown_connections 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 --2- 172.21.15.70:0/3776574846 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c071d10 0x7f315c0d0690 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 -- 172.21.15.70:0/3776574846 >> 172.21.15.70:0/3776574846 conn(0x7f315c06be00 msgr2=0x7f315c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 -- 172.21.15.70:0/3776574846 shutdown_connections 2024-04-15T15:16:22.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.667+0000 7f3162057640 1 -- 172.21.15.70:0/3776574846 wait complete. 2024-04-15T15:16:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.668+0000 7f3162057640 1 Processor -- start 2024-04-15T15:16:22.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.668+0000 7f3162057640 1 -- start start 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 0x7f315c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f315c0d0bd0 0x7f315c157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f315c15d860 con 0x7f315c072d80 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f315c15d580 con 0x7f315c071d10 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f3162057640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f315c0afaf0 con 0x7f315c0d0bd0 2024-04-15T15:16:22.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49638/0 (socket says 172.21.15.70:49638) 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 -- 172.21.15.70:0/2134232493 learned_addr learned my addr 172.21.15.70:0/2134232493 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315b7fe640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 0x7f315c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315bfff640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f315c0d0bd0 0x7f315c157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 -- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f315c0d0bd0 msgr2=0x7f315c157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f315c0d0bd0 0x7f315c157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 -- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 msgr2=0x7f315c15bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 0x7f315c15bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315affd640 1 -- 172.21.15.70:0/2134232493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f315c15ad00 con 0x7f315c072d80 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.669+0000 7f315b7fe640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 0x7f315c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f315affd640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3150037040 tx=0x7f3150004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150039050 con 0x7f315c072d80 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f315003a040 con 0x7f315c072d80 2024-04-15T15:16:22.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f315c15afe0 con 0x7f315c072d80 2024-04-15T15:16:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150003bd0 con 0x7f315c072d80 2024-04-15T15:16:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.670+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f315c15b5a0 con 0x7f315c072d80 2024-04-15T15:16:22.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.671+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f315c0689e0 con 0x7f315c072d80 2024-04-15T15:16:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.671+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f3150047030 con 0x7f315c072d80 2024-04-15T15:16:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.671+0000 7f3143fff640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 0x7f314806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.672+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f31500adf00 con 0x7f315c072d80 2024-04-15T15:16:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.672+0000 7f315b7fe640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 0x7f314806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.675 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.672+0000 7f315b7fe640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 0x7f314806edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f314c000f60 tx=0x7f314c007c50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.676+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f315004e020 con 0x7f315c072d80 2024-04-15T15:16:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.851+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f315c0aff10 con 0x7f315c072d80 2024-04-15T15:16:22.853 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.851+0000 7f3143fff640 1 -- 172.21.15.70:0/2134232493 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f315007c580 con 0x7f315c072d80 2024-04-15T15:16:22.854 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:22.855 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 msgr2=0x7f314806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 0x7f314806edb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f314c000f60 tx=0x7f314c007c50 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 msgr2=0x7f315c157250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f3150037040 tx=0x7f3150004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 shutdown_connections 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f315c0d0bd0 0x7f315c157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f314806c8f0 0x7f314806edb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f315c072d80 0x7f315c157250 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 --2- 172.21.15.70:0/2134232493 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f315c071d10 0x7f315c15bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.853+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 >> 172.21.15.70:0/2134232493 conn(0x7f315c06be00 msgr2=0x7f315c0736d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.854+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 shutdown_connections 2024-04-15T15:16:22.856 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.854+0000 7f3162057640 1 -- 172.21.15.70:0/2134232493 wait complete. 2024-04-15T15:16:22.868 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-15T15:16:22.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.968+0000 7fd65577c640 1 Processor -- start 2024-04-15T15:16:22.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 -- start start 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009a9d0 0x7fd65009adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd65015ebb0 0x7fd65006cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd65009b2f0 con 0x7fd65009b780 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd6500cd8c0 con 0x7fd65015ebb0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.969+0000 7fd65577c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd6500ced60 con 0x7fd65009a9d0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64e7fc640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd65015ebb0 0x7fd65006cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd65009b780 0x7fd6500b76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38826/0 (socket says 172.21.15.70:38826) 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64f7fe640 1 -- 172.21.15.70:0/3001895642 learned_addr learned my addr 172.21.15.70:0/3001895642 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64effd640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009a9d0 0x7fd65009adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64e7fc640 1 -- 172.21.15.70:0/3001895642 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009a9d0 msgr2=0x7fd65009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64e7fc640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009a9d0 0x7fd65009adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64e7fc640 1 -- 172.21.15.70:0/3001895642 >> v1:172.21.15.70:6789/0 conn(0x7fd65009b780 legacy=0x7fd6500b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64dffb640 1 -- 172.21.15.70:0/3001895642 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954294341 0 0) 0x7fd65009b2f0 con 0x7fd65009b780 2024-04-15T15:16:22.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.970+0000 7fd64e7fc640 1 -- 172.21.15.70:0/3001895642 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6500cccb0 con 0x7fd65015ebb0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd64e7fc640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd65015ebb0 0x7fd65006cd00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd6380097a0 tx=0x7fd63802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bb8a6e480c8903b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd64dffb640 1 -- 172.21.15.70:0/3001895642 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638035020 con 0x7fd65015ebb0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd64dffb640 1 -- 172.21.15.70:0/3001895642 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638031830 con 0x7fd65015ebb0 2024-04-15T15:16:22.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd64dffb640 1 -- 172.21.15.70:0/3001895642 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638031c00 con 0x7fd65015ebb0 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 -- 172.21.15.70:0/3001895642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd65015ebb0 msgr2=0x7fd65006cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd65015ebb0 0x7fd65006cd00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd6380097a0 tx=0x7fd63802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 -- 172.21.15.70:0/3001895642 shutdown_connections 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd65015ebb0 0x7fd65006cd00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 --2- 172.21.15.70:0/3001895642 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009a9d0 0x7fd65009adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 -- 172.21.15.70:0/3001895642 >> 172.21.15.70:0/3001895642 conn(0x7fd6500b3fb0 msgr2=0x7fd6500b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 -- 172.21.15.70:0/3001895642 shutdown_connections 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.971+0000 7fd65577c640 1 -- 172.21.15.70:0/3001895642 wait complete. 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 Processor -- start 2024-04-15T15:16:22.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 -- start start 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 0x7fd65015a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 0x7fd6500c8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6500cd550 con 0x7fd6500c8a60 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6500cd270 con 0x7fd65009a9d0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.973+0000 7fd65577c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd65006d240 con 0x7fd65009b780 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49664/0 (socket says 172.21.15.70:49664) 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 -- 172.21.15.70:0/1555709185 learned_addr learned my addr 172.21.15.70:0/1555709185 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64e7fc640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 0x7fd6500c8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.978 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64effd640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 0x7fd65015a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 -- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 msgr2=0x7fd6500c8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 0x7fd6500c8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 -- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 msgr2=0x7fd65015a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 0x7fd65015a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 -- 172.21.15.70:0/1555709185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6500c6de0 con 0x7fd6500c8a60 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64e7fc640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 0x7fd6500c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64effd640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 0x7fd65015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd64f7fe640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd64800c480 tx=0x7fd64800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648018070 con 0x7fd6500c8a60 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6480074e0 con 0x7fd6500c8a60 2024-04-15T15:16:22.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6500c70c0 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd648004a20 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.974+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6500c7680 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.975+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6500690f0 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.976+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd64801b070 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.976+0000 7fd6477fe640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 0x7fd63006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.976+0000 7fd64effd640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 0x7fd63006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.976+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fd64808ac10 con 0x7fd6500c8a60 2024-04-15T15:16:22.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.977+0000 7fd64effd640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 0x7fd63006edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd640007f50 tx=0x7fd6400074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:22.982 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:22.980+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd648059290 con 0x7fd6500c8a60 2024-04-15T15:16:23.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.155+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fd6500751d0 con 0x7fd6500c8a60 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.156+0000 7fd6477fe640 1 -- 172.21.15.70:0/1555709185 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7fd648058b90 con 0x7fd6500c8a60 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout:[ 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout: "poolnum": 1, 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout: "poolname": ".mgr" 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:23.158 INFO:teuthology.orchestra.run.smithi070.stdout:] 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 msgr2=0x7fd63006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 0x7fd63006edb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd640007f50 tx=0x7fd6400074a0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 msgr2=0x7fd6500c60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd64800c480 tx=0x7fd64800c940 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 shutdown_connections 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd63006c8f0 0x7fd63006edb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6500c8a60 0x7fd6500c60d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd65009b780 0x7fd6500c8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 --2- 172.21.15.70:0/1555709185 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd65009a9d0 0x7fd65015a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 >> 172.21.15.70:0/1555709185 conn(0x7fd6500b3fb0 msgr2=0x7fd6500cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 shutdown_connections 2024-04-15T15:16:23.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.158+0000 7fd65577c640 1 -- 172.21.15.70:0/1555709185 wait complete. 2024-04-15T15:16:23.170 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-15T15:16:23.171 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-15T15:16:23.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 Processor -- start 2024-04-15T15:16:23.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 -- start start 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bd80b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd80b1a20 con 0x7f8bd81626e0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd80d0fe0 con 0x7f8bd815eb70 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.276+0000 7f8bde3e9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd80d0d90 con 0x7f8bd80b1520 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bdc986640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8bd81626e0 0x7f8bd806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38830/0 (socket says 172.21.15.70:38830) 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bdc986640 1 -- 172.21.15.70:0/3249520656 learned_addr learned my addr 172.21.15.70:0/3249520656 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bd80b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1309574461 0 0) 0x7f8bd80b1a20 con 0x7f8bd81626e0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bc0003520 con 0x7f8bd81626e0 2024-04-15T15:16:23.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3274400117 0 0) 0x7f8bd80d0fe0 con 0x7f8bd815eb70 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8bd80b1a20 con 0x7f8bd815eb70 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 591676194 0 0) 0x7f8bc0003520 con 0x7f8bd81626e0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 msgr2=0x7f8bd80b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bd80b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 >> v1:172.21.15.96:6789/0 conn(0x7f8bd815eb70 legacy=0x7f8bd809b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd80d12c0 con 0x7f8bd81626e0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f8bd00019f0 con 0x7f8bd81626e0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 >> v1:172.21.15.70:6789/0 conn(0x7f8bd81626e0 legacy=0x7f8bd806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bc0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bc0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bc0004d40 0x7f8bc00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd80d0d90 con 0x7f8bc00040d0 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc000bb50 con 0x7f8bc0004d40 2024-04-15T15:16:23.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8bc000bd20 con 0x7f8bd80b1520 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bc0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bd7fff640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bc0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.277+0000 7f8bdc986640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bc0004d40 0x7f8bc00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 -- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 msgr2=0x7f8bc0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b1520 0x7f8bc0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 -- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bc0004d40 msgr2=0x7f8bc00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bc0004d40 0x7f8bc00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 -- 172.21.15.70:0/3249520656 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc0003520 con 0x7f8bc00040d0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bc0004800 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009ac0 tx=0x7f8bc0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afae2a85681d5c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc039070 con 0x7f8bc00040d0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bcc02c6b0 con 0x7f8bc00040d0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.278+0000 7f8bd6ffd640 1 -- 172.21.15.70:0/3249520656 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bcc009690 con 0x7f8bc00040d0 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 -- 172.21.15.70:0/3249520656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 msgr2=0x7f8bc0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/3249520656 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bc0004800 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc009ac0 tx=0x7f8bc0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:23.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bdc986640 1 -- 172.21.15.70:0/3249520656 reap_dead start 2024-04-15T15:16:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 -- 172.21.15.70:0/3249520656 shutdown_connections 2024-04-15T15:16:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 -- 172.21.15.70:0/3249520656 >> 172.21.15.70:0/3249520656 conn(0x7f8bd80b3f70 msgr2=0x7f8bd80b4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 -- 172.21.15.70:0/3249520656 shutdown_connections 2024-04-15T15:16:23.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.279+0000 7f8bde3e9640 1 -- 172.21.15.70:0/3249520656 wait complete. 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.280+0000 7f8bde3e9640 1 Processor -- start 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.280+0000 7f8bde3e9640 1 -- start start 2024-04-15T15:16:23.287 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b37f0 0x7f8bd80b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 0x7f8bd815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd80b4a90 con 0x7f8bc00040d0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd815ff60 con 0x7f8bd815bc90 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bde3e9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd80b4c10 con 0x7f8bd80b37f0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bd7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49674/0 (socket says 172.21.15.70:49674) 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.281+0000 7f8bd7fff640 1 -- 172.21.15.70:0/1746447269 learned_addr learned my addr 172.21.15.70:0/1746447269 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 -- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b37f0 msgr2=0x7f8bd80b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bdc986640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 0x7f8bd815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b37f0 0x7f8bd80b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b37f0 0x7f8bd80b27a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 -- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 msgr2=0x7f8bd815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 0x7f8bd815c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 -- 172.21.15.70:0/1746447269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd80cb310 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bdc986640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 0x7f8bd815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd7fff640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800f7a0 tx=0x7f8bc800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800fe50 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc8018070 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc800d730 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd80b2ce0 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.282+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd80c98d0 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.283+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd80689e0 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.283+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8bc800d8d0 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.284+0000 7f8bd4ff9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 0x7f8bc406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.284+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 0x7f8bc406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.284+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f8bc808b890 con 0x7f8bc00040d0 2024-04-15T15:16:23.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.284+0000 7f8bd77fe640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 0x7f8bc406ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc036000 tx=0x7f8bcc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.290 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.288+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8bd80b2ce0 con 0x7f8bc00040d0 2024-04-15T15:16:23.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.463+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8bd80b8400 con 0x7f8bc00040d0 2024-04-15T15:16:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.863+0000 7f8bd4ff9640 1 -- 172.21.15.70:0/1746447269 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7f8bc801a310 con 0x7f8bc00040d0 2024-04-15T15:16:23.865 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' created 2024-04-15T15:16:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 msgr2=0x7f8bc406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 0x7f8bc406ede0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8bcc036000 tx=0x7f8bcc030000 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 msgr2=0x7f8bd8157a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800f7a0 tx=0x7f8bc800fc60 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 shutdown_connections 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8bd815bc90 0x7f8bd815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.865+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8bd80b37f0 0x7f8bd80b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.866+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bc406c920 0x7f8bc406ede0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.866+0000 7f8bde3e9640 1 --2- 172.21.15.70:0/1746447269 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8bc00040d0 0x7f8bd8157a00 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:23.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.866+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 >> 172.21.15.70:0/1746447269 conn(0x7f8bd80b3f70 msgr2=0x7f8bd80aff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.866+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 shutdown_connections 2024-04-15T15:16:23.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.866+0000 7f8bde3e9640 1 -- 172.21.15.70:0/1746447269 wait complete. 2024-04-15T15:16:23.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-15T15:16:23.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 Processor -- start 2024-04-15T15:16:23.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 -- start start 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 0x7f432c070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432c06ff30 con 0x7f432c0ce8e0 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f432c0cecc0 con 0x7f432c09b0e0 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.993+0000 7f4333242640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f432c070820 con 0x7f432c071d90 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f4330fde640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f432c0ce8e0 0x7f432c09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38836/0 (socket says 172.21.15.70:38836) 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f4330fde640 1 -- 172.21.15.70:0/1605971409 learned_addr learned my addr 172.21.15.70:0/1605971409 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:23.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f43317df640 1 --2- 172.21.15.70:0/1605971409 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 0x7f432c070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22779515 0 0) 0x7f432c06ff30 con 0x7f432c0ce8e0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f431c003520 con 0x7f432c0ce8e0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1130333207 0 0) 0x7f432c0cecc0 con 0x7f432c09b0e0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f432c06ff30 con 0x7f432c09b0e0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2407107488 0 0) 0x7f431c003520 con 0x7f432c0ce8e0 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f43317df640 1 -- 172.21.15.70:0/1605971409 >> v1:172.21.15.96:6789/0 conn(0x7f432c09b0e0 legacy=0x7f432c0cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f43317df640 1 -- 172.21.15.70:0/1605971409 >> v1:172.21.15.70:6789/0 conn(0x7f432c0ce8e0 legacy=0x7f432c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.994+0000 7f43317df640 1 -- 172.21.15.70:0/1605971409 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c09b9d0 con 0x7f432c071d90 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.995+0000 7f43317df640 1 --2- 172.21.15.70:0/1605971409 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 0x7f432c070cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4324009dc0 tx=0x7f432402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aecad0744f6f311 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.995+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324039070 con 0x7f432c071d90 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.995+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f432402c880 con 0x7f432c071d90 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.995+0000 7f432b7fe640 1 -- 172.21.15.70:0/1605971409 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4320001670 con 0x7f432c071d90 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 -- 172.21.15.70:0/1605971409 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 msgr2=0x7f432c070cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 --2- 172.21.15.70:0/1605971409 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 0x7f432c070cb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4324009dc0 tx=0x7f432402bd50 comp rx=0 tx=0).stop 2024-04-15T15:16:23.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 -- 172.21.15.70:0/1605971409 shutdown_connections 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 --2- 172.21.15.70:0/1605971409 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c071d90 0x7f432c070cb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 -- 172.21.15.70:0/1605971409 >> 172.21.15.70:0/1605971409 conn(0x7f432c0b3f40 msgr2=0x7f432c0b4350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 -- 172.21.15.70:0/1605971409 shutdown_connections 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.996+0000 7f4333242640 1 -- 172.21.15.70:0/1605971409 wait complete. 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.997+0000 7f4333242640 1 Processor -- start 2024-04-15T15:16:24.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.997+0000 7f4333242640 1 -- start start 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f432c071d90 0x7f432c1cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c09b0e0 0x7f432c0cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f432c0704b0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f432c0701d0 con 0x7f432c071d90 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4333242640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f432c071970 con 0x7f432c09b0e0 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49676/0 (socket says 172.21.15.70:49676) 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 -- 172.21.15.70:0/3054249690 learned_addr learned my addr 172.21.15.70:0/3054249690 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f432bfff640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c09b0e0 0x7f432c0cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f4330fde640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f432c071d90 0x7f432c1cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 -- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c09b0e0 msgr2=0x7f432c0cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c09b0e0 0x7f432c0cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 -- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f432c071d90 msgr2=0x7f432c1cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f432c071d90 0x7f432c1cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 -- 172.21.15.70:0/3054249690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f432c0cbdf0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.998+0000 7f43317df640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4324009ee0 tx=0x7f4324004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.999+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324039050 con 0x7f432c0ce8e0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.999+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4324045070 con 0x7f432c0ce8e0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.999+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f432c0c98c0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.999+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4324009450 con 0x7f432c0ce8e0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:23.999+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f432c1d0150 con 0x7f432c0ce8e0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.000+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f432c0689e0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.000+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f4324040690 con 0x7f432c0ce8e0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.001+0000 7f43297fa640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 0x7f431006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.001+0000 7f4330fde640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 0x7f431006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.001+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f43240ae170 con 0x7f432c0ce8e0 2024-04-15T15:16:24.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.001+0000 7f4330fde640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 0x7f431006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4320000ff0 tx=0x7f4320001150 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:24.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.005+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f432407bfa0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.181+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f432c0b34d0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.867+0000 7f43297fa640 1 -- 172.21.15.70:0/3054249690 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f43240802a0 con 0x7f432c0ce8e0 2024-04-15T15:16:24.870 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' created 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.869+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 msgr2=0x7f431006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.869+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 0x7f431006edb0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4320000ff0 tx=0x7f4320001150 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.869+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 msgr2=0x7f432c0cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.869+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4324009ee0 tx=0x7f4324004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 shutdown_connections 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f431006c8f0 0x7f431006edb0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f432c0ce8e0 0x7f432c0cb740 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f432c09b0e0 0x7f432c0cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 --2- 172.21.15.70:0/3054249690 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f432c071d90 0x7f432c1cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 >> 172.21.15.70:0/3054249690 conn(0x7f432c0b3f40 msgr2=0x7f432c1de010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 shutdown_connections 2024-04-15T15:16:24.873 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:24.870+0000 7f4333242640 1 -- 172.21.15.70:0/3054249690 wait complete. 2024-04-15T15:16:24.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-15T15:16:25.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.001+0000 7f6648247640 1 Processor -- start 2024-04-15T15:16:25.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.001+0000 7f6648247640 1 -- start start 2024-04-15T15:16:25.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6648247640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f66400701f0 0x7f6640160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6648247640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6640070720 con 0x7f664009a9d0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6648247640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6640070a90 con 0x7f664015fb30 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6648247640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f664009ba90 con 0x7f66400701f0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6645fe3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f664009a9d0 0x7f664009adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38846/0 (socket says 172.21.15.70:38846) 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.002+0000 7f6645fe3640 1 -- 172.21.15.70:0/1379712472 learned_addr learned my addr 172.21.15.70:0/1379712472 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3065567466 0 0) 0x7f6640070720 con 0x7f664009a9d0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f662c003520 con 0x7f664009a9d0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 --2- 172.21.15.70:0/1379712472 >> v2:172.21.15.96:3300/0 conn(0x7f66400701f0 0x7f6640160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 873082750 0 0) 0x7f662c003520 con 0x7f664009a9d0 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 >> v2:172.21.15.96:3300/0 conn(0x7f66400701f0 msgr2=0x7f6640160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 --2- 172.21.15.70:0/1379712472 >> v2:172.21.15.96:3300/0 conn(0x7f66400701f0 0x7f6640160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 >> v1:172.21.15.96:6789/0 conn(0x7f664015fb30 legacy=0x7f66400d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6640070d70 con 0x7f664009a9d0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f663c0019f0 con 0x7f664009a9d0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 >> v1:172.21.15.70:6789/0 conn(0x7f664009a9d0 legacy=0x7f664009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f662c0039b0 0x7f662c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 0x7f662c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c0051c0 0x7f662c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f662c00c0e0 con 0x7f662c004800 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f662c00c290 con 0x7f662c0039b0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f662c00c490 con 0x7f662c0051c0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 0x7f662c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f6645fe3640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f662c0039b0 0x7f662c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66467e4640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c0051c0 0x7f662c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 -- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c0051c0 msgr2=0x7f662c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c0051c0 0x7f662c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 -- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f662c0039b0 msgr2=0x7f662c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f662c0039b0 0x7f662c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66457e2640 1 -- 172.21.15.70:0/1379712472 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c003520 con 0x7f662c004800 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.003+0000 7f66467e4640 1 -- 172.21.15.70:0/1379712472 reap_dead start 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.004+0000 7f66457e2640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 0x7f662c004c80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f663000ba00 tx=0x7f662c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3fec95168d4e82 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.004+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6630036020 con 0x7f662c004800 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.004+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6630032620 con 0x7f662c004800 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.004+0000 7f6644fe1640 1 -- 172.21.15.70:0/1379712472 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f663000b600 con 0x7f662c004800 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 -- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 msgr2=0x7f662c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 0x7f662c004c80 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f663000ba00 tx=0x7f662c005830 comp rx=0 tx=0).stop 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 -- 172.21.15.70:0/1379712472 shutdown_connections 2024-04-15T15:16:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 --2- 172.21.15.70:0/1379712472 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f662c004800 0x7f662c004c80 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 -- 172.21.15.70:0/1379712472 >> 172.21.15.70:0/1379712472 conn(0x7f66400d2460 msgr2=0x7f66400d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 -- 172.21.15.70:0/1379712472 shutdown_connections 2024-04-15T15:16:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.005+0000 7f6648247640 1 -- 172.21.15.70:0/1379712472 wait complete. 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.006+0000 7f6648247640 1 Processor -- start 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.006+0000 7f6648247640 1 -- start start 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c004800 0x7f66401f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f664014f550 0x7f664009c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f664015ffc0 con 0x7f664014f9d0 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66400d08c0 con 0x7f664014f550 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f66400b1c60 con 0x7f662c004800 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49684/0 (socket says 172.21.15.70:49684) 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 -- 172.21.15.70:0/139630258 learned_addr learned my addr 172.21.15.70:0/139630258 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:25.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66457e2640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f664014f550 0x7f664009c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6645fe3640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c004800 0x7f66401f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 -- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c004800 msgr2=0x7f66401f58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c004800 0x7f66401f58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 -- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f664014f550 msgr2=0x7f664009c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f664014f550 0x7f664009c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 -- 172.21.15.70:0/139630258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6640152540 con 0x7f664014f9d0 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66467e4640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f66400c94e0 tx=0x7f663800e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6638019070 con 0x7f664014f9d0 2024-04-15T15:16:25.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f663800ee40 con 0x7f664014f9d0 2024-04-15T15:16:25.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66400c7020 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.007+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66380035f0 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.008+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66400c75e0 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.009+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66400689e0 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.009+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f663801c070 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.010+0000 7f66367fc640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 0x7f662406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.010+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7f663808ae30 con 0x7f664014f9d0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.010+0000 7f6645fe3640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 0x7f662406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:25.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.010+0000 7f6645fe3640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 0x7f662406edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f663c001330 tx=0x7f663c006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:25.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.014+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6638058b70 con 0x7f664014f9d0 2024-04-15T15:16:25.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.226+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f66400c9120 con 0x7f664014f9d0 2024-04-15T15:16:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.869+0000 7f66367fc640 1 -- 172.21.15.70:0/139630258 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7f663805ce70 con 0x7f664014f9d0 2024-04-15T15:16:25.874 INFO:teuthology.orchestra.run.smithi070.stderr:new fs with metadata pool 5 and data pool 6 2024-04-15T15:16:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.873+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 msgr2=0x7f662406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.873+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 0x7f662406edb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f663c001330 tx=0x7f663c006040 comp rx=0 tx=0).stop 2024-04-15T15:16:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.873+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 msgr2=0x7f664009c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:25.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.873+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f66400c94e0 tx=0x7f663800e610 comp rx=0 tx=0).stop 2024-04-15T15:16:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 shutdown_connections 2024-04-15T15:16:25.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f662406c8f0 0x7f662406edb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f664014f9d0 0x7f664009c910 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f664014f550 0x7f664009c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 --2- 172.21.15.70:0/139630258 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f662c004800 0x7f66401f58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:25.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.875+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 >> 172.21.15.70:0/139630258 conn(0x7f66400d2460 msgr2=0x7f664015e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.883+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 shutdown_connections 2024-04-15T15:16:25.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:25.884+0000 7f6648247640 1 -- 172.21.15.70:0/139630258 wait complete. 2024-04-15T15:16:25.908 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:26.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.011+0000 7fda00047640 1 Processor -- start 2024-04-15T15:16:26.013 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.011+0000 7fda00047640 1 -- start start 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fda00047640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd9f80b1620 0x7fd9f80b1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fda00047640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f80af920 con 0x7fd9f815ecb0 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fda00047640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f80afc00 con 0x7fd9f80af340 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fda00047640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f815f090 con 0x7fd9f80b1620 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fd9fdde3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd9f815ecb0 0x7fd9f80b0b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38860/0 (socket says 172.21.15.70:38860) 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fd9fdde3640 1 -- 172.21.15.70:0/4244770549 learned_addr learned my addr 172.21.15.70:0/4244770549 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fd9fe5e4640 1 --2- 172.21.15.70:0/4244770549 >> v2:172.21.15.96:3300/0 conn(0x7fd9f80b1620 0x7fd9f80b1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979087576 0 0) 0x7fd9f80af920 con 0x7fd9f815ecb0 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.012+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9e4003520 con 0x7fd9f815ecb0 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3392631736 0 0) 0x7fd9f80afc00 con 0x7fd9f80af340 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9f80af920 con 0x7fd9f80af340 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2523376390 0 0) 0x7fd9e4003520 con 0x7fd9f815ecb0 2024-04-15T15:16:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/4244770549 >> v1:172.21.15.96:6789/0 conn(0x7fd9f80af340 legacy=0x7fd9f80b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/4244770549 >> v1:172.21.15.70:6789/0 conn(0x7fd9f815ecb0 legacy=0x7fd9f80b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.013+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/4244770549 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f80afee0 con 0x7fd9f80b1620 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fd9fe5e4640 1 --2- 172.21.15.70:0/4244770549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f80b1ab0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009d90 tx=0x7fd9f002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f4c14909a849715 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0039070 con 0x7fd9f80b1620 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f002c9d0 con 0x7fd9f80b1620 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fd9fcde1640 1 -- 172.21.15.70:0/4244770549 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f4001670 con 0x7fd9f80b1620 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 -- 172.21.15.70:0/4244770549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 msgr2=0x7fd9f80b1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 --2- 172.21.15.70:0/4244770549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f80b1ab0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0009d90 tx=0x7fd9f002bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 -- 172.21.15.70:0/4244770549 shutdown_connections 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 --2- 172.21.15.70:0/4244770549 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f80b1ab0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 -- 172.21.15.70:0/4244770549 >> 172.21.15.70:0/4244770549 conn(0x7fd9f80b6870 msgr2=0x7fd9f80b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 -- 172.21.15.70:0/4244770549 shutdown_connections 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.014+0000 7fda00047640 1 -- 172.21.15.70:0/4244770549 wait complete. 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.015+0000 7fda00047640 1 Processor -- start 2024-04-15T15:16:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 -- start start 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9f80af340 0x7fd9f80cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f8161ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80ce7a0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80ce620 con 0x7fd9f80b1620 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fda00047640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f80ce920 con 0x7fd9f80af340 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fd9fe5e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fd9fe5e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49692/0 (socket says 172.21.15.70:49692) 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.016+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/2167830057 learned_addr learned my addr 172.21.15.70:0/2167830057 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9f80af340 msgr2=0x7fd9f80cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9f80af340 0x7fd9f80cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 msgr2=0x7fd9f8161ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f8161ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 -- 172.21.15.70:0/2167830057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f8161ff0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9fe5e4640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0037040 tx=0x7fd9f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0039050 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9f0045070 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9f0009450 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f80cbd90 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.017+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f80cc630 con 0x7fd9f815ecb0 2024-04-15T15:16:26.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.018+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f80689e0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.019+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd9f0040690 con 0x7fd9f815ecb0 2024-04-15T15:16:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.019+0000 7fd9ee7fc640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 0x7fd9dc06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.019+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fd9f00ae3b0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.019+0000 7fd9fdde3640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 0x7fd9dc06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.020+0000 7fd9fdde3640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 0x7fd9dc06edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4005880 tx=0x7fd9f4001230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.024+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd9f004f020 con 0x7fd9f815ecb0 2024-04-15T15:16:26.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.202+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9f806c7c0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.203+0000 7fd9ee7fc640 1 -- 172.21.15.70:0/2167830057 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16895 (secure 0 0 0) 0x7fd9f007c6e0 con 0x7fd9f815ecb0 2024-04-15T15:16:26.205 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:26.206 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:25.228640+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 msgr2=0x7fd9dc06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 0x7fd9dc06edb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd9f4005880 tx=0x7fd9f4001230 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 msgr2=0x7fd9f81f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0037040 tx=0x7fd9f0004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 shutdown_connections 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd9dc06c8f0 0x7fd9dc06edb0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd9f815ecb0 0x7fd9f81f3bc0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd9f80b1620 0x7fd9f8161ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 --2- 172.21.15.70:0/2167830057 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd9f80af340 0x7fd9f80cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 >> 172.21.15.70:0/2167830057 conn(0x7fd9f80b6870 msgr2=0x7fd9f80b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 shutdown_connections 2024-04-15T15:16:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.206+0000 7fda00047640 1 -- 172.21.15.70:0/2167830057 wait complete. 2024-04-15T15:16:26.219 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.320+0000 7fb147a4a640 1 Processor -- start 2024-04-15T15:16:26.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.320+0000 7fb147a4a640 1 -- start start 2024-04-15T15:16:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb147a4a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb140071d10 0x7fb1400d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb147a4a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14015db40 con 0x7fb1400d0bd0 2024-04-15T15:16:26.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb147a4a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb140072170 con 0x7fb140072d80 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb145fe7640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb1400d0bd0 0x7fb14006e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38862/0 (socket says 172.21.15.70:38862) 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb147a4a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb1400d1040 con 0x7fb140071d10 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb145fe7640 1 -- 172.21.15.70:0/3702627192 learned_addr learned my addr 172.21.15.70:0/3702627192 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb144fe5640 1 --2- 172.21.15.70:0/3702627192 >> v2:172.21.15.96:3300/0 conn(0x7fb140071d10 0x7fb1400d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4030607558 0 0) 0x7fb14015db40 con 0x7fb1400d0bd0 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.321+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb12c003520 con 0x7fb1400d0bd0 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2512224195 0 0) 0x7fb140072170 con 0x7fb140072d80 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb14015db40 con 0x7fb140072d80 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb144fe5640 1 -- 172.21.15.70:0/3702627192 >> v1:172.21.15.96:6789/0 conn(0x7fb140072d80 legacy=0x7fb1400af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 329704661 0 0) 0x7fb12c003520 con 0x7fb1400d0bd0 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb144fe5640 1 -- 172.21.15.70:0/3702627192 >> v1:172.21.15.70:6789/0 conn(0x7fb1400d0bd0 legacy=0x7fb14006e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb144fe5640 1 -- 172.21.15.70:0/3702627192 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14015cfc0 con 0x7fb140071d10 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.322+0000 7fb144fe5640 1 --2- 172.21.15.70:0/3702627192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1400d0690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1300097a0 tx=0x7fb13002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a3e498e35a6f454 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb130037070 con 0x7fb140071d10 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb130030510 con 0x7fb140071d10 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb137fff640 1 -- 172.21.15.70:0/3702627192 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138001670 con 0x7fb140071d10 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 -- 172.21.15.70:0/3702627192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 msgr2=0x7fb1400d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 --2- 172.21.15.70:0/3702627192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1400d0690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb1300097a0 tx=0x7fb13002b790 comp rx=0 tx=0).stop 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 -- 172.21.15.70:0/3702627192 shutdown_connections 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 --2- 172.21.15.70:0/3702627192 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1400d0690 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 -- 172.21.15.70:0/3702627192 >> 172.21.15.70:0/3702627192 conn(0x7fb14006be00 msgr2=0x7fb14006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 -- 172.21.15.70:0/3702627192 shutdown_connections 2024-04-15T15:16:26.326 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.323+0000 7fb147a4a640 1 -- 172.21.15.70:0/3702627192 wait complete. 2024-04-15T15:16:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 Processor -- start 2024-04-15T15:16:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 -- start start 2024-04-15T15:16:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1401503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb140072d80 0x7fb1400cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb14015d860 con 0x7fb1400d0bd0 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb14015d580 con 0x7fb140071d10 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb147a4a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb1400afaf0 con 0x7fb140072d80 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb145fe7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb145fe7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49706/0 (socket says 172.21.15.70:49706) 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.325+0000 7fb145fe7640 1 -- 172.21.15.70:0/669987609 learned_addr learned my addr 172.21.15.70:0/669987609 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb1457e6640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1401503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb144fe5640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb140072d80 0x7fb1400cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 -- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb140072d80 msgr2=0x7fb1400cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb140072d80 0x7fb1400cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 -- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 msgr2=0x7fb1401503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1401503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 -- 172.21.15.70:0/669987609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb140150920 con 0x7fb1400d0bd0 2024-04-15T15:16:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb145fe7640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c520 tx=0x7fb13800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138017070 con 0x7fb1400d0bd0 2024-04-15T15:16:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb138005ce0 con 0x7fb1400d0bd0 2024-04-15T15:16:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1401f3020 con 0x7fb1400d0bd0 2024-04-15T15:16:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb138007270 con 0x7fb1400d0bd0 2024-04-15T15:16:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.326+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1401f35e0 con 0x7fb1400d0bd0 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.327+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1400689e0 con 0x7fb1400d0bd0 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.327+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fb13801a070 con 0x7fb1400d0bd0 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.328+0000 7fb135ffb640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 0x7fb12406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.328+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7fb13808b120 con 0x7fb1400d0bd0 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.328+0000 7fb1457e6640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 0x7fb12406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.331 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.328+0000 7fb1457e6640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 0x7fb12406edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb13c001560 tx=0x7fb13c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.332+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb1380593d0 con 0x7fb1400d0bd0 2024-04-15T15:16:26.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.510+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb14006cd10 con 0x7fb1400d0bd0 2024-04-15T15:16:26.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.511+0000 7fb135ffb640 1 -- 172.21.15.70:0/669987609 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16895 (secure 0 0 0) 0x7fb138058cd0 con 0x7fb1400d0bd0 2024-04-15T15:16:26.513 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:26.513 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":38,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:25.228640+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 msgr2=0x7fb12406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 0x7fb12406edb0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb13c001560 tx=0x7fb13c009040 comp rx=0 tx=0).stop 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 msgr2=0x7fb1400b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb13800c520 tx=0x7fb13800c9e0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 shutdown_connections 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb12406c8f0 0x7fb12406edb0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1400d0bd0 0x7fb1400b10e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb140072d80 0x7fb1400cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 --2- 172.21.15.70:0/669987609 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb140071d10 0x7fb1401503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 >> 172.21.15.70:0/669987609 conn(0x7fb14006be00 msgr2=0x7fb14015b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 shutdown_connections 2024-04-15T15:16:26.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.513+0000 7fb147a4a640 1 -- 172.21.15.70:0/669987609 wait complete. 2024-04-15T15:16:26.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-15T15:16:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.629+0000 7f01dff82640 1 Processor -- start 2024-04-15T15:16:26.631 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.630+0000 7f01dff82640 1 -- start start 2024-04-15T15:16:26.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.630+0000 7f01dff82640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01d80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.630+0000 7f01dff82640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d8067e30 con 0x7f01d815ec10 2024-04-15T15:16:26.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.630+0000 7f01dff82640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d80b3c10 con 0x7f01d80b53c0 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.630+0000 7f01dff82640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f01d80b3ef0 con 0x7f01d815e830 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dd51d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f01d815ec10 0x7f01d8067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38866/0 (socket says 172.21.15.70:38866) 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dd51d640 1 -- 172.21.15.70:0/1965636696 learned_addr learned my addr 172.21.15.70:0/1965636696 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01d80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2240624310 0 0) 0x7f01d8067e30 con 0x7f01d815ec10 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01c0003520 con 0x7f01d815ec10 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393042329 0 0) 0x7f01d80b3c10 con 0x7f01d80b53c0 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01d8067e30 con 0x7f01d80b53c0 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1843997077 0 0) 0x7f01c0003520 con 0x7f01d815ec10 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 msgr2=0x7f01d80b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01d80b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 >> v1:172.21.15.96:6789/0 conn(0x7f01d80b53c0 legacy=0x7f01d80aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d80b41d0 con 0x7f01d815ec10 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f01cc0019f0 con 0x7f01d815ec10 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01d80b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 >> v1:172.21.15.70:6789/0 conn(0x7f01d815ec10 legacy=0x7f01d8067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01c0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004100 0x7f01c0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c0004d40 0x7f01c00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01d80b3ef0 con 0x7f01c0004d40 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01c000bad0 con 0x7f01c0004100 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f01c000bcd0 con 0x7f01d815e830 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01de51f640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c0004d40 0x7f01c00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01c0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.631+0000 7f01dd51d640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004100 0x7f01c0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 -- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 msgr2=0x7f01c0003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d815e830 0x7f01c0003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 -- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004100 msgr2=0x7f01c0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004100 0x7f01c0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 -- 172.21.15.70:0/1965636696 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01c0003520 con 0x7f01c0004d40 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01de51f640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c0004d40 0x7f01c00051c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f01c80018a0 tx=0x7f01c0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0b1b425e2b5661 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8037070 con 0x7f01c0004d40 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c80052d0 con 0x7f01c0004d40 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.632+0000 7f01dcd1c640 1 -- 172.21.15.70:0/1965636696 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c80014f0 con 0x7f01c0004d40 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 -- 172.21.15.70:0/1965636696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c0004d40 msgr2=0x7f01c00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 --2- 172.21.15.70:0/1965636696 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01c0004d40 0x7f01c00051c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f01c80018a0 tx=0x7f01c0005830 comp rx=0 tx=0).stop 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01de51f640 1 -- 172.21.15.70:0/1965636696 reap_dead start 2024-04-15T15:16:26.638 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 -- 172.21.15.70:0/1965636696 shutdown_connections 2024-04-15T15:16:26.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 -- 172.21.15.70:0/1965636696 >> 172.21.15.70:0/1965636696 conn(0x7f01d8073360 msgr2=0x7f01d806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 -- 172.21.15.70:0/1965636696 shutdown_connections 2024-04-15T15:16:26.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.633+0000 7f01dff82640 1 -- 172.21.15.70:0/1965636696 wait complete. 2024-04-15T15:16:26.641 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.634+0000 7f01dff82640 1 Processor -- start 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.634+0000 7f01dff82640 1 -- start start 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004d40 0x7f01d815b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d81f4750 0x7f01d81589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f01d815ff60 con 0x7f01d8158f10 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01d80b4790 con 0x7f01c0004d40 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dff82640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f01d81600e0 con 0x7f01d81f4750 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49718/0 (socket says 172.21.15.70:49718) 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 -- 172.21.15.70:0/2616428417 learned_addr learned my addr 172.21.15.70:0/2616428417 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004d40 0x7f01d815b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.642 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01dd51d640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d81f4750 0x7f01d81589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 -- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d81f4750 msgr2=0x7f01d81589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d81f4750 0x7f01d81589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 -- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004d40 msgr2=0x7f01d815b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004d40 0x7f01d815b610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.635+0000 7f01de51f640 1 -- 172.21.15.70:0/2616428417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d81e5110 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01de51f640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f01c8005040 tx=0x7f01c80012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8045070 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c8038040 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d815bb50 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c8037070 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.636+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d815d570 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.637+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d80689e0 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.637+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f01c8048070 con 0x7f01d8158f10 2024-04-15T15:16:26.643 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.638+0000 7f01d67fc640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 0x7f01c406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.638+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f01c80ac660 con 0x7f01d8158f10 2024-04-15T15:16:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.638+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 0x7f01c406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.638+0000 7f01ddd1e640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 0x7f01c406ede0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009bb0 tx=0x7f01d0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:26.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.642+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f01c807a2b0 con 0x7f01d8158f10 2024-04-15T15:16:26.857 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.855+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f01d806ff10 con 0x7f01d8158f10 2024-04-15T15:16:26.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.881+0000 7f01d67fc640 1 -- 172.21.15.70:0/2616428417 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f01c80aae20 con 0x7f01d8158f10 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 msgr2=0x7f01c406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 0x7f01c406ede0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f01d0009bb0 tx=0x7f01d0009190 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 msgr2=0x7f01d8159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f01c8005040 tx=0x7f01c80012d0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 shutdown_connections 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f01c406c920 0x7f01c406ede0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01d8158f10 0x7f01d8159390 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01d81f4750 0x7f01d81589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 --2- 172.21.15.70:0/2616428417 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f01c0004d40 0x7f01d815b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:26.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 >> 172.21.15.70:0/2616428417 conn(0x7f01d8073360 msgr2=0x7f01d8067720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:26.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 shutdown_connections 2024-04-15T15:16:26.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:26.884+0000 7f01dff82640 1 -- 172.21.15.70:0/2616428417 wait complete. 2024-04-15T15:16:26.896 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-15T15:16:26.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-15T15:16:27.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.001+0000 7f9fa9aba640 1 Processor -- start 2024-04-15T15:16:27.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.001+0000 7f9fa9aba640 1 -- start start 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa9aba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa40701f0 0x7f9fa4160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa9aba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fa415fb30 0x7f9fa40d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa9aba640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fa4070720 con 0x7f9fa409a9d0 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa9aba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f9fa409ba90 con 0x7f9fa415fb30 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa9aba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa40d08c0 con 0x7f9fa40701f0 2024-04-15T15:16:27.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fa415fb30 0x7f9fa40d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9fa409a9d0 0x7f9fa409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38880/0 (socket says 172.21.15.70:38880) 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f9fa415fb30 0x7f9fa40d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49720/0 (socket says 172.21.15.70:49720) 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa37fe640 1 -- 172.21.15.70:0/1454907918 learned_addr learned my addr 172.21.15.70:0/1454907918 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa2ffd640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa40701f0 0x7f9fa4160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 -- 172.21.15.70:0/1454907918 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa40701f0 msgr2=0x7f9fa4160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa40701f0 0x7f9fa4160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 -- 172.21.15.70:0/1454907918 >> v1:172.21.15.70:6789/0 conn(0x7f9fa409a9d0 legacy=0x7f9fa409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa3fff640 1 -- 172.21.15.70:0/1454907918 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa4070a90 con 0x7f9fa415fb30 2024-04-15T15:16:27.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.002+0000 7f9fa27fc640 1 -- 172.21.15.70:0/1454907918 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2913862514 0 0) 0x7f9fa4070720 con 0x7f9fa409a9d0 2024-04-15T15:16:27.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.003+0000 7f9fa3fff640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40d0d50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0097d0 tx=0x7f9f9c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8838120a11b8af server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.003+0000 7f9fa27fc640 1 -- 172.21.15.70:0/1454907918 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c035020 con 0x7f9fa415fb30 2024-04-15T15:16:27.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.003+0000 7f9fa27fc640 1 -- 172.21.15.70:0/1454907918 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f9c031800 con 0x7f9fa415fb30 2024-04-15T15:16:27.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.003+0000 7f9fa27fc640 1 -- 172.21.15.70:0/1454907918 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c031bd0 con 0x7f9fa415fb30 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 -- 172.21.15.70:0/1454907918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 msgr2=0x7f9fa40d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40d0d50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0097d0 tx=0x7f9f9c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 -- 172.21.15.70:0/1454907918 shutdown_connections 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40d0d50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/1454907918 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa40701f0 0x7f9fa4160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 -- 172.21.15.70:0/1454907918 >> 172.21.15.70:0/1454907918 conn(0x7f9fa40d2460 msgr2=0x7f9fa40d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 -- 172.21.15.70:0/1454907918 shutdown_connections 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.004+0000 7f9fa9aba640 1 -- 172.21.15.70:0/1454907918 wait complete. 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.005+0000 7f9fa9aba640 1 Processor -- start 2024-04-15T15:16:27.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.005+0000 7f9fa9aba640 1 -- start start 2024-04-15T15:16:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40701f0 0x7f9fa4154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 0x7f9fa40c9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa415ffc0 con 0x7f9fa415fb30 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9fa4071050 con 0x7f9fa40701f0 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa9aba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9fa40b1c60 con 0x7f9fa409a9d0 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49728/0 (socket says 172.21.15.70:49728) 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 -- 172.21.15.70:0/4032284992 learned_addr learned my addr 172.21.15.70:0/4032284992 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa37fe640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40701f0 0x7f9fa4154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa2ffd640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 0x7f9fa40c9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 -- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 msgr2=0x7f9fa40c9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 0x7f9fa40c9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 -- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40701f0 msgr2=0x7f9fa4154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40701f0 0x7f9fa4154b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 -- 172.21.15.70:0/4032284992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fa40b2a20 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa2ffd640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 0x7f9fa40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.006+0000 7f9fa3fff640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0098f0 tx=0x7f9f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.007+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c041070 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.007+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f9c03f070 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.007+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fa40b2d00 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.007+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f9c035040 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.007+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fa40b0580 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.008+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fa40689e0 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.008+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9f9c038020 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.009+0000 7f9f8bfff640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 0x7f9f9006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.009+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f9f9c0ad880 con 0x7f9fa415fb30 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.009+0000 7f9fa37fe640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 0x7f9f9006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:27.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.009+0000 7f9fa37fe640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 0x7f9f9006edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f940012e0 tx=0x7f9f94001700 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:27.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.013+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9f9c07bb20 con 0x7f9fa415fb30 2024-04-15T15:16:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.225+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f9fa40753c0 con 0x7f9fa415fb30 2024-04-15T15:16:27.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.887+0000 7f9f8bfff640 1 -- 172.21.15.70:0/4032284992 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f9f9c07b420 con 0x7f9fa415fb30 2024-04-15T15:16:27.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 msgr2=0x7f9f9006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 0x7f9f9006edb0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9f940012e0 tx=0x7f9f94001700 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 msgr2=0x7f9fa40c8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9f9c0098f0 tx=0x7f9f9c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 shutdown_connections 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9f9006c8f0 0x7f9f9006edb0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9fa415fb30 0x7f9fa40c8330 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9fa409a9d0 0x7f9fa40c9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 --2- 172.21.15.70:0/4032284992 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9fa40701f0 0x7f9fa4154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 >> 172.21.15.70:0/4032284992 conn(0x7f9fa40d2460 msgr2=0x7f9fa4161ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 shutdown_connections 2024-04-15T15:16:27.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:27.889+0000 7f9fa9aba640 1 -- 172.21.15.70:0/4032284992 wait complete. 2024-04-15T15:16:27.903 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-15T15:16:27.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.008+0000 7fea770a5640 1 Processor -- start 2024-04-15T15:16:28.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.008+0000 7fea770a5640 1 -- start start 2024-04-15T15:16:28.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea770a5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fea7015ecb0 0x7fea700b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea770a5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 0x7fea700b1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea770a5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea700b10d0 con 0x7fea700b1b60 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea770a5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fea7015f090 con 0x7fea7015ecb0 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea770a5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fea700cdf80 con 0x7fea700b4350 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea74e41640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fea7015ecb0 0x7fea700b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea74e41640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fea7015ecb0 0x7fea700b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49742/0 (socket says 172.21.15.70:49742) 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.009+0000 7fea74e41640 1 -- 172.21.15.70:0/4045464917 learned_addr learned my addr 172.21.15.70:0/4045464917 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea74e41640 1 -- 172.21.15.70:0/4045464917 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 msgr2=0x7fea700b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea6ffff640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 0x7fea700b1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea74e41640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 0x7fea700b1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea74e41640 1 -- 172.21.15.70:0/4045464917 >> v1:172.21.15.96:6789/0 conn(0x7fea700b1b60 legacy=0x7fea700ce410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea74e41640 1 -- 172.21.15.70:0/4045464917 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea700af920 con 0x7fea7015ecb0 2024-04-15T15:16:28.012 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea6ffff640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 0x7fea700b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.010+0000 7fea74e41640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7015ecb0 0x7fea700b0b90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fea60009e50 tx=0x7fea6002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f7951e8ce30ecb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea6f7fe640 1 -- 172.21.15.70:0/4045464917 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6002cb00 con 0x7fea7015ecb0 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea6f7fe640 1 -- 172.21.15.70:0/4045464917 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea6002cca0 con 0x7fea7015ecb0 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea6f7fe640 1 -- 172.21.15.70:0/4045464917 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60034610 con 0x7fea7015ecb0 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 -- 172.21.15.70:0/4045464917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7015ecb0 msgr2=0x7fea700b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7015ecb0 0x7fea700b0b90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fea60009e50 tx=0x7fea6002bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 -- 172.21.15.70:0/4045464917 shutdown_connections 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea700b4350 0x7fea700b1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.014 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 --2- 172.21.15.70:0/4045464917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea7015ecb0 0x7fea700b0b90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 -- 172.21.15.70:0/4045464917 >> 172.21.15.70:0/4045464917 conn(0x7fea700b6870 msgr2=0x7fea700b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 -- 172.21.15.70:0/4045464917 shutdown_connections 2024-04-15T15:16:28.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.011+0000 7fea770a5640 1 -- 172.21.15.70:0/4045464917 wait complete. 2024-04-15T15:16:28.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.012+0000 7fea770a5640 1 Processor -- start 2024-04-15T15:16:28.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.012+0000 7fea770a5640 1 -- start start 2024-04-15T15:16:28.016 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea700b4350 0x7fea700d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea7015ecb0 0x7fea700ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fea700b4790 con 0x7fea700b1b60 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fea700afee0 con 0x7fea700b4350 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea770a5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fea700b4910 con 0x7fea7015ecb0 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea74e41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea74e41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49744/0 (socket says 172.21.15.70:49744) 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.013+0000 7fea74e41640 1 -- 172.21.15.70:0/34503770 learned_addr learned my addr 172.21.15.70:0/34503770 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea6ffff640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea700b4350 0x7fea700d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea75642640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea7015ecb0 0x7fea700ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 -- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea7015ecb0 msgr2=0x7fea700ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea7015ecb0 0x7fea700ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 -- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea700b4350 msgr2=0x7fea700d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea700b4350 0x7fea700d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 -- 172.21.15.70:0/34503770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea700d0af0 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea74e41640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fea60000f80 tx=0x7fea60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea60043020 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea6002ce50 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea700d0dd0 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6003e690 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.014+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea7015fae0 con 0x7fea700b1b60 2024-04-15T15:16:28.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.015+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea700753d0 con 0x7fea700b1b60 2024-04-15T15:16:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.016+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fea60046020 con 0x7fea700b1b60 2024-04-15T15:16:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.016+0000 7fea6d7fa640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 0x7fea5006ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.016+0000 7fea6ffff640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 0x7fea5006ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.016+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fea600ae490 con 0x7fea700b1b60 2024-04-15T15:16:28.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.017+0000 7fea6ffff640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 0x7fea5006ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea700d1ad0 tx=0x7fea58008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.021+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fea700753d0 con 0x7fea700b1b60 2024-04-15T15:16:28.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.236+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea700689e0 con 0x7fea700b1b60 2024-04-15T15:16:28.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.237+0000 7fea6d7fa640 1 -- 172.21.15.70:0/34503770 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7fea6004d030 con 0x7fea700b1b60 2024-04-15T15:16:28.239 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:28.239 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:16:27.226372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/2334246723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2334246723},{"type":"v1","addr":"172.21.15.96:6835","nonce":2334246723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4907,"qdb_cluster":[4907]},"id":2}]} 2024-04-15T15:16:28.239 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 15 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 msgr2=0x7fea5006ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 0x7fea5006ede0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea700d1ad0 tx=0x7fea58008040 comp rx=0 tx=0).stop 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 msgr2=0x7fea7014cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fea60000f80 tx=0x7fea60004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 shutdown_connections 2024-04-15T15:16:28.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fea7015ecb0 0x7fea700ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fea700b4350 0x7fea700d0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fea5006c920 0x7fea5006ede0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 --2- 172.21.15.70:0/34503770 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fea700b1b60 0x7fea7014cd10 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 >> 172.21.15.70:0/34503770 conn(0x7fea700b6870 msgr2=0x7fea700b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 shutdown_connections 2024-04-15T15:16:28.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.239+0000 7fea770a5640 1 -- 172.21.15.70:0/34503770 wait complete. 2024-04-15T15:16:28.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.356+0000 7f075f888640 1 Processor -- start 2024-04-15T15:16:28.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.357+0000 7f075f888640 1 -- start start 2024-04-15T15:16:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.357+0000 7f075f888640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07580d0690 0x7f07580d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.357+0000 7f075f888640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f075815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075f888640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07580d0fb0 con 0x7f075815ef70 2024-04-15T15:16:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075f888640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f075815cb50 con 0x7f07580d0690 2024-04-15T15:16:28.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075f888640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f075815bfd0 con 0x7f07580b3160 2024-04-15T15:16:28.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075d624640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f07580d0690 0x7f07580d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075ce23640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f075815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075de25640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f075815ef70 0x7f075815f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:56874/0 (socket says 172.21.15.70:56874) 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075de25640 1 -- 172.21.15.70:0/672840344 learned_addr learned my addr 172.21.15.70:0/672840344 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075d624640 1 -- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 msgr2=0x7f075815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075d624640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f075815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075d624640 1 -- 172.21.15.70:0/672840344 >> v1:172.21.15.96:6789/0 conn(0x7f075815ef70 legacy=0x7f075815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075d624640 1 -- 172.21.15.70:0/672840344 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075815c2b0 con 0x7f07580d0690 2024-04-15T15:16:28.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.358+0000 7f075ce23640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f075815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.359+0000 7f075d624640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07580d0690 0x7f07580d0a70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0748009d90 tx=0x7f074802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c81150b2314fc729 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.359+0000 7f0757fff640 1 -- 172.21.15.70:0/672840344 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748039070 con 0x7f07580d0690 2024-04-15T15:16:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f0757fff640 1 -- 172.21.15.70:0/672840344 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074802cc50 con 0x7f07580d0690 2024-04-15T15:16:28.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f0757fff640 1 -- 172.21.15.70:0/672840344 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0748034440 con 0x7f07580d0690 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 -- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07580d0690 msgr2=0x7f07580d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07580d0690 0x7f07580d0a70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f0748009d90 tx=0x7f074802bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 -- 172.21.15.70:0/672840344 shutdown_connections 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f075815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 --2- 172.21.15.70:0/672840344 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f07580d0690 0x7f07580d0a70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 -- 172.21.15.70:0/672840344 >> 172.21.15.70:0/672840344 conn(0x7f075806be00 msgr2=0x7f075806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 -- 172.21.15.70:0/672840344 shutdown_connections 2024-04-15T15:16:28.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.360+0000 7f075f888640 1 -- 172.21.15.70:0/672840344 wait complete. 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.361+0000 7f075f888640 1 Processor -- start 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.361+0000 7f075f888640 1 -- start start 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f0758155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f075815ef70 0x7f075809af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f07580b4fb0 con 0x7f07580d0690 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f075815c870 con 0x7f075815ef70 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075f888640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f075815bbd0 con 0x7f07580b3160 2024-04-15T15:16:28.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075ce23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075ce23640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49748/0 (socket says 172.21.15.70:49748) 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.362+0000 7f075ce23640 1 -- 172.21.15.70:0/864848260 learned_addr learned my addr 172.21.15.70:0/864848260 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075d624640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f0758155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075de25640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f075815ef70 0x7f075809af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 -- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 msgr2=0x7f0758155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f0758155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 -- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f075815ef70 msgr2=0x7f075809af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f075815ef70 0x7f075809af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 -- 172.21.15.70:0/864848260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07580cede0 con 0x7f07580d0690 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075ce23640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f074c00e960 tx=0x7f074c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c017070 con 0x7f07580d0690 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f074c00ca00 con 0x7f07580d0690 2024-04-15T15:16:28.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f074c00cdd0 con 0x7f07580d0690 2024-04-15T15:16:28.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07580cf0c0 con 0x7f07580d0690 2024-04-15T15:16:28.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.363+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07580ccc80 con 0x7f07580d0690 2024-04-15T15:16:28.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.364+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07580cd0c0 con 0x7f07580d0690 2024-04-15T15:16:28.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.365+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f074c01a070 con 0x7f07580d0690 2024-04-15T15:16:28.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.365+0000 7f0755ffb640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 0x7f074406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.365+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f074c08bf40 con 0x7f07580d0690 2024-04-15T15:16:28.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.365+0000 7f075d624640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 0x7f074406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.366+0000 7f075d624640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 0x7f074406edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f0748030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.370+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f074c05a1f0 con 0x7f07580d0690 2024-04-15T15:16:28.547 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.545+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f07580689e0 con 0x7f07580d0690 2024-04-15T15:16:28.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.547+0000 7f0755ffb640 1 -- 172.21.15.70:0/864848260 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16882 (secure 0 0 0) 0x7f074c059af0 con 0x7f07580d0690 2024-04-15T15:16:28.549 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:28.549 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:26.878225+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 msgr2=0x7f074406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 0x7f074406edb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0748036000 tx=0x7f0748030000 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 msgr2=0x7f075809dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f074c00e960 tx=0x7f074c00ee20 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 shutdown_connections 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f075815ef70 0x7f075809af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f074406c8f0 0x7f074406edb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f07580d0690 0x7f075809dae0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 --2- 172.21.15.70:0/864848260 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f07580b3160 0x7f0758155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 >> 172.21.15.70:0/864848260 conn(0x7f075806be00 msgr2=0x7f075806e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 shutdown_connections 2024-04-15T15:16:28.551 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.549+0000 7f075f888640 1 -- 172.21.15.70:0/864848260 wait complete. 2024-04-15T15:16:28.562 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-15T15:16:28.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:28.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.662+0000 7f009ccc8640 1 Processor -- start 2024-04-15T15:16:28.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.662+0000 7f009ccc8640 1 -- start start 2024-04-15T15:16:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009ccc8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f009815e700 0x7f00980b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009ccc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f00980d2360 0x7f00980d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009ccc8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00980b47a0 con 0x7f009815eae0 2024-04-15T15:16:28.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009ccc8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f00980b53b0 con 0x7f009815e700 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009ccc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f00980b4a80 con 0x7f00980d2360 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f009659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f009815e700 0x7f00980b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.663+0000 7f0096d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f009815eae0 0x7f00980cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38882/0 (socket says 172.21.15.70:38882) 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f0096d9d640 1 -- 172.21.15.70:0/1335090093 learned_addr learned my addr 172.21.15.70:0/1335090093 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009659c640 1 -- 172.21.15.70:0/1335090093 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f00980d2360 msgr2=0x7f00980d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009659c640 1 --2- 172.21.15.70:0/1335090093 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f00980d2360 0x7f00980d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009659c640 1 -- 172.21.15.70:0/1335090093 >> v1:172.21.15.70:6789/0 conn(0x7f009815eae0 legacy=0x7f00980cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009659c640 1 -- 172.21.15.70:0/1335090093 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00980b4d60 con 0x7f009815e700 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009559a640 1 -- 172.21.15.70:0/1335090093 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385618964 0 0) 0x7f00980b47a0 con 0x7f009815eae0 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.664+0000 7f009659c640 1 --2- 172.21.15.70:0/1335090093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815e700 0x7f00980b5840 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0088009790 tx=0x7f008802b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de87193d74fb3906 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009559a640 1 -- 172.21.15.70:0/1335090093 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088035020 con 0x7f009815e700 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009559a640 1 -- 172.21.15.70:0/1335090093 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0088031800 con 0x7f009815e700 2024-04-15T15:16:28.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009559a640 1 -- 172.21.15.70:0/1335090093 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088031bd0 con 0x7f009815e700 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 -- 172.21.15.70:0/1335090093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815e700 msgr2=0x7f00980b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 --2- 172.21.15.70:0/1335090093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815e700 0x7f00980b5840 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f0088009790 tx=0x7f008802b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 -- 172.21.15.70:0/1335090093 shutdown_connections 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 --2- 172.21.15.70:0/1335090093 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f00980d2360 0x7f00980d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 --2- 172.21.15.70:0/1335090093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815e700 0x7f00980b5840 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 -- 172.21.15.70:0/1335090093 >> 172.21.15.70:0/1335090093 conn(0x7f009808bf60 msgr2=0x7f00980d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 -- 172.21.15.70:0/1335090093 shutdown_connections 2024-04-15T15:16:28.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.665+0000 7f009ccc8640 1 -- 172.21.15.70:0/1335090093 wait complete. 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 Processor -- start 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 -- start start 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 0x7f00980b9b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 0x7f00980ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f00980b5d80 con 0x7f009815eae0 2024-04-15T15:16:28.672 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f00980b4450 con 0x7f00980d2360 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f009ccc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f00980d1d40 con 0x7f009815e700 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.667+0000 7f0096d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49774/0 (socket says 172.21.15.70:49774) 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 -- 172.21.15.70:0/793390479 learned_addr learned my addr 172.21.15.70:0/793390479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 -- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 msgr2=0x7f00980ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f009659c640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 0x7f00980b9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0095d9b640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 0x7f00980ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 0x7f00980ca3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 -- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 msgr2=0x7f00980b9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 0x7f00980b9b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 -- 172.21.15.70:0/793390479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00980ba040 con 0x7f009815eae0 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0095d9b640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 0x7f00980ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f009659c640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 0x7f00980b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0096d9d640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f009000c700 tx=0x7f009000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0090018070 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f009000ce10 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00980b81e0 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f009000a860 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.668+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00980b87a0 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.669+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00980689e0 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.670+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f009001b070 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.670+0000 7f0086ffd640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 0x7f007806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.670+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f009008b1c0 con 0x7f009815eae0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.670+0000 7f009659c640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 0x7f007806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:28.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.671+0000 7f009659c640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 0x7f007806edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0088036000 tx=0x7f0088008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:28.676 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.674+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00900594f0 con 0x7f009815eae0 2024-04-15T15:16:28.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.856+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f00980c7f50 con 0x7f007806c8f0 2024-04-15T15:16:28.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.862+0000 7f0086ffd640 1 -- 172.21.15.70:0/793390479 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249848 (secure 0 0 0) 0x7f0088031940 con 0x7f007806c8f0 2024-04-15T15:16:28.868 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:28.869 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 msgr2=0x7f007806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 0x7f007806edb0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0088036000 tx=0x7f0088008030 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 msgr2=0x7f009815c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f009000c700 tx=0x7f009000cbc0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 shutdown_connections 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f007806c8f0 0x7f007806edb0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f009815eae0 0x7f009815c9c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f009815e700 0x7f00980ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 --2- 172.21.15.70:0/793390479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00980d2360 0x7f00980b9b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 >> 172.21.15.70:0/793390479 conn(0x7f009808bf60 msgr2=0x7f009806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 shutdown_connections 2024-04-15T15:16:28.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:28.867+0000 7f009ccc8640 1 -- 172.21.15.70:0/793390479 wait complete. 2024-04-15T15:16:28.881 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-15T15:16:28.085226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":191,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":166,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237408,"kb_used_data":4596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4706304,"data_stored":2665647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.219711"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884133+0000","last_change":"2024-04-15T15:16:25.883054+0000","last_active":"2024-04-15T15:16:26.884133+0000","last_peered":"2024-04-15T15:16:26.884133+0000","last_clean":"2024-04-15T15:16:26.884133+0000","last_became_active":"2024-04-15T15:16:25.882744+0000","last_became_peered":"2024-04-15T15:16:25.882744+0000","last_unstale":"2024-04-15T15:16:26.884133+0000","last_undegraded":"2024-04-15T15:16:26.884133+0000","last_fullsized":"2024-04-15T15:16:26.884133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:06:55.204459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883810+0000","last_change":"2024-04-15T15:16:25.888729+0000","last_active":"2024-04-15T15:16:26.883810+0000","last_peered":"2024-04-15T15:16:26.883810+0000","last_clean":"2024-04-15T15:16:26.883810+0000","last_became_active":"2024-04-15T15:16:25.888450+0000","last_became_peered":"2024-04-15T15:16:25.888450+0000","last_unstale":"2024-04-15T15:16:26.883810+0000","last_undegraded":"2024-04-15T15:16:26.883810+0000","last_fullsized":"2024-04-15T15:16:26.883810+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902977+0000","last_change":"2024-04-15T15:16:25.902269+0000","last_active":"2024-04-15T15:16:25.902977+0000","last_peered":"2024-04-15T15:16:25.902977+0000","last_clean":"2024-04-15T15:16:25.902977+0000","last_became_active":"2024-04-15T15:16:25.901972+0000","last_became_peered":"2024-04-15T15:16:25.901972+0000","last_unstale":"2024-04-15T15:16:25.902977+0000","last_undegraded":"2024-04-15T15:16:25.902977+0000","last_fullsized":"2024-04-15T15:16:25.902977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898887+0000","last_change":"2024-04-15T15:16:25.897033+0000","last_active":"2024-04-15T15:16:25.898887+0000","last_peered":"2024-04-15T15:16:25.898887+0000","last_clean":"2024-04-15T15:16:25.898887+0000","last_became_active":"2024-04-15T15:16:25.896432+0000","last_became_peered":"2024-04-15T15:16:25.896432+0000","last_unstale":"2024-04-15T15:16:25.898887+0000","last_undegraded":"2024-04-15T15:16:25.898887+0000","last_fullsized":"2024-04-15T15:16:25.898887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:19.565523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891565+0000","last_change":"2024-04-15T15:16:25.889069+0000","last_active":"2024-04-15T15:16:25.891565+0000","last_peered":"2024-04-15T15:16:25.891565+0000","last_clean":"2024-04-15T15:16:25.891565+0000","last_became_active":"2024-04-15T15:16:25.888793+0000","last_became_peered":"2024-04-15T15:16:25.888793+0000","last_unstale":"2024-04-15T15:16:25.891565+0000","last_undegraded":"2024-04-15T15:16:25.891565+0000","last_fullsized":"2024-04-15T15:16:25.891565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:48.611396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888862+0000","last_change":"2024-04-15T15:16:25.887072+0000","last_active":"2024-04-15T15:16:25.888862+0000","last_peered":"2024-04-15T15:16:25.888862+0000","last_clean":"2024-04-15T15:16:25.888862+0000","last_became_active":"2024-04-15T15:16:25.886821+0000","last_became_peered":"2024-04-15T15:16:25.886821+0000","last_unstale":"2024-04-15T15:16:25.888862+0000","last_undegraded":"2024-04-15T15:16:25.888862+0000","last_fullsized":"2024-04-15T15:16:25.888862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904107+0000","last_change":"2024-04-15T15:16:25.900378+0000","last_active":"2024-04-15T15:16:25.904107+0000","last_peered":"2024-04-15T15:16:25.904107+0000","last_clean":"2024-04-15T15:16:25.904107+0000","last_became_active":"2024-04-15T15:16:25.900071+0000","last_became_peered":"2024-04-15T15:16:25.900071+0000","last_unstale":"2024-04-15T15:16:25.904107+0000","last_undegraded":"2024-04-15T15:16:25.904107+0000","last_fullsized":"2024-04-15T15:16:25.904107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:43.288974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883618+0000","last_change":"2024-04-15T15:16:25.893507+0000","last_active":"2024-04-15T15:16:26.883618+0000","last_peered":"2024-04-15T15:16:26.883618+0000","last_clean":"2024-04-15T15:16:26.883618+0000","last_became_active":"2024-04-15T15:16:25.893005+0000","last_became_peered":"2024-04-15T15:16:25.893005+0000","last_unstale":"2024-04-15T15:16:26.883618+0000","last_undegraded":"2024-04-15T15:16:26.883618+0000","last_fullsized":"2024-04-15T15:16:26.883618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.895340+0000","last_change":"2024-04-15T15:16:25.894870+0000","last_active":"2024-04-15T15:16:25.895340+0000","last_peered":"2024-04-15T15:16:25.895340+0000","last_clean":"2024-04-15T15:16:25.895340+0000","last_became_active":"2024-04-15T15:16:25.894343+0000","last_became_peered":"2024-04-15T15:16:25.894343+0000","last_unstale":"2024-04-15T15:16:25.895340+0000","last_undegraded":"2024-04-15T15:16:25.895340+0000","last_fullsized":"2024-04-15T15:16:25.895340+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904248+0000","last_change":"2024-04-15T15:16:25.900279+0000","last_active":"2024-04-15T15:16:25.904248+0000","last_peered":"2024-04-15T15:16:25.904248+0000","last_clean":"2024-04-15T15:16:25.904248+0000","last_became_active":"2024-04-15T15:16:25.899969+0000","last_became_peered":"2024-04-15T15:16:25.899969+0000","last_unstale":"2024-04-15T15:16:25.904248+0000","last_undegraded":"2024-04-15T15:16:25.904248+0000","last_fullsized":"2024-04-15T15:16:25.904248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:19:11.196336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911723+0000","last_change":"2024-04-15T15:16:25.904761+0000","last_active":"2024-04-15T15:16:25.911723+0000","last_peered":"2024-04-15T15:16:25.911723+0000","last_clean":"2024-04-15T15:16:25.911723+0000","last_became_active":"2024-04-15T15:16:25.904477+0000","last_became_peered":"2024-04-15T15:16:25.904477+0000","last_unstale":"2024-04-15T15:16:25.911723+0000","last_undegraded":"2024-04-15T15:16:25.911723+0000","last_fullsized":"2024-04-15T15:16:25.911723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884502+0000","last_change":"2024-04-15T15:16:25.882646+0000","last_active":"2024-04-15T15:16:25.884502+0000","last_peered":"2024-04-15T15:16:25.884502+0000","last_clean":"2024-04-15T15:16:25.884502+0000","last_became_active":"2024-04-15T15:16:25.882375+0000","last_became_peered":"2024-04-15T15:16:25.882375+0000","last_unstale":"2024-04-15T15:16:25.884502+0000","last_undegraded":"2024-04-15T15:16:25.884502+0000","last_fullsized":"2024-04-15T15:16:25.884502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906634+0000","last_change":"2024-04-15T15:16:25.901003+0000","last_active":"2024-04-15T15:16:25.906634+0000","last_peered":"2024-04-15T15:16:25.906634+0000","last_clean":"2024-04-15T15:16:25.906634+0000","last_became_active":"2024-04-15T15:16:25.900729+0000","last_became_peered":"2024-04-15T15:16:25.900729+0000","last_unstale":"2024-04-15T15:16:25.906634+0000","last_undegraded":"2024-04-15T15:16:25.906634+0000","last_fullsized":"2024-04-15T15:16:25.906634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:34.769819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885365+0000","last_change":"2024-04-15T15:16:25.899260+0000","last_active":"2024-04-15T15:16:26.885365+0000","last_peered":"2024-04-15T15:16:26.885365+0000","last_clean":"2024-04-15T15:16:26.885365+0000","last_became_active":"2024-04-15T15:16:25.898979+0000","last_became_peered":"2024-04-15T15:16:25.898979+0000","last_unstale":"2024-04-15T15:16:26.885365+0000","last_undegraded":"2024-04-15T15:16:26.885365+0000","last_fullsized":"2024-04-15T15:16:26.885365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904064+0000","last_change":"2024-04-15T15:16:25.899744+0000","last_active":"2024-04-15T15:16:25.904064+0000","last_peered":"2024-04-15T15:16:25.904064+0000","last_clean":"2024-04-15T15:16:25.904064+0000","last_became_active":"2024-04-15T15:16:25.899472+0000","last_became_peered":"2024-04-15T15:16:25.899472+0000","last_unstale":"2024-04-15T15:16:25.904064+0000","last_undegraded":"2024-04-15T15:16:25.904064+0000","last_fullsized":"2024-04-15T15:16:25.904064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885281+0000","last_change":"2024-04-15T15:16:25.892123+0000","last_active":"2024-04-15T15:16:26.885281+0000","last_peered":"2024-04-15T15:16:26.885281+0000","last_clean":"2024-04-15T15:16:26.885281+0000","last_became_active":"2024-04-15T15:16:25.891853+0000","last_became_peered":"2024-04-15T15:16:25.891853+0000","last_unstale":"2024-04-15T15:16:26.885281+0000","last_undegraded":"2024-04-15T15:16:26.885281+0000","last_fullsized":"2024-04-15T15:16:26.885281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884688+0000","last_change":"2024-04-15T15:16:25.880039+0000","last_active":"2024-04-15T15:16:25.884688+0000","last_peered":"2024-04-15T15:16:25.884688+0000","last_clean":"2024-04-15T15:16:25.884688+0000","last_became_active":"2024-04-15T15:16:25.879753+0000","last_became_peered":"2024-04-15T15:16:25.879753+0000","last_unstale":"2024-04-15T15:16:25.884688+0000","last_undegraded":"2024-04-15T15:16:25.884688+0000","last_fullsized":"2024-04-15T15:16:25.884688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:58:15.988048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885269+0000","last_change":"2024-04-15T15:16:25.897291+0000","last_active":"2024-04-15T15:16:26.885269+0000","last_peered":"2024-04-15T15:16:26.885269+0000","last_clean":"2024-04-15T15:16:26.885269+0000","last_became_active":"2024-04-15T15:16:25.896977+0000","last_became_peered":"2024-04-15T15:16:25.896977+0000","last_unstale":"2024-04-15T15:16:26.885269+0000","last_undegraded":"2024-04-15T15:16:26.885269+0000","last_fullsized":"2024-04-15T15:16:26.885269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901153+0000","last_change":"2024-04-15T15:16:25.889177+0000","last_active":"2024-04-15T15:16:25.901153+0000","last_peered":"2024-04-15T15:16:25.901153+0000","last_clean":"2024-04-15T15:16:25.901153+0000","last_became_active":"2024-04-15T15:16:25.888843+0000","last_became_peered":"2024-04-15T15:16:25.888843+0000","last_unstale":"2024-04-15T15:16:25.901153+0000","last_undegraded":"2024-04-15T15:16:25.901153+0000","last_fullsized":"2024-04-15T15:16:25.901153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885679+0000","last_change":"2024-04-15T15:16:25.892453+0000","last_active":"2024-04-15T15:16:26.885679+0000","last_peered":"2024-04-15T15:16:26.885679+0000","last_clean":"2024-04-15T15:16:26.885679+0000","last_became_active":"2024-04-15T15:16:25.892168+0000","last_became_peered":"2024-04-15T15:16:25.892168+0000","last_unstale":"2024-04-15T15:16:26.885679+0000","last_undegraded":"2024-04-15T15:16:26.885679+0000","last_fullsized":"2024-04-15T15:16:26.885679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891390+0000","last_change":"2024-04-15T15:16:25.885032+0000","last_active":"2024-04-15T15:16:25.891390+0000","last_peered":"2024-04-15T15:16:25.891390+0000","last_clean":"2024-04-15T15:16:25.891390+0000","last_became_active":"2024-04-15T15:16:25.884397+0000","last_became_peered":"2024-04-15T15:16:25.884397+0000","last_unstale":"2024-04-15T15:16:25.891390+0000","last_undegraded":"2024-04-15T15:16:25.891390+0000","last_fullsized":"2024-04-15T15:16:25.891390+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:31.155865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885450+0000","last_change":"2024-04-15T15:16:25.882933+0000","last_active":"2024-04-15T15:16:26.885450+0000","last_peered":"2024-04-15T15:16:26.885450+0000","last_clean":"2024-04-15T15:16:26.885450+0000","last_became_active":"2024-04-15T15:16:25.882643+0000","last_became_peered":"2024-04-15T15:16:25.882643+0000","last_unstale":"2024-04-15T15:16:26.885450+0000","last_undegraded":"2024-04-15T15:16:26.885450+0000","last_fullsized":"2024-04-15T15:16:26.885450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891210+0000","last_change":"2024-04-15T15:16:25.890339+0000","last_active":"2024-04-15T15:16:25.891210+0000","last_peered":"2024-04-15T15:16:25.891210+0000","last_clean":"2024-04-15T15:16:25.891210+0000","last_became_active":"2024-04-15T15:16:25.889922+0000","last_became_peered":"2024-04-15T15:16:25.889922+0000","last_unstale":"2024-04-15T15:16:25.891210+0000","last_undegraded":"2024-04-15T15:16:25.891210+0000","last_fullsized":"2024-04-15T15:16:25.891210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885230+0000","last_change":"2024-04-15T15:16:25.890968+0000","last_active":"2024-04-15T15:16:26.885230+0000","last_peered":"2024-04-15T15:16:26.885230+0000","last_clean":"2024-04-15T15:16:26.885230+0000","last_became_active":"2024-04-15T15:16:25.890686+0000","last_became_peered":"2024-04-15T15:16:25.890686+0000","last_unstale":"2024-04-15T15:16:26.885230+0000","last_undegraded":"2024-04-15T15:16:26.885230+0000","last_fullsized":"2024-04-15T15:16:26.885230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904156+0000","last_change":"2024-04-15T15:16:25.900260+0000","last_active":"2024-04-15T15:16:25.904156+0000","last_peered":"2024-04-15T15:16:25.904156+0000","last_clean":"2024-04-15T15:16:25.904156+0000","last_became_active":"2024-04-15T15:16:25.899948+0000","last_became_peered":"2024-04-15T15:16:25.899948+0000","last_unstale":"2024-04-15T15:16:25.904156+0000","last_undegraded":"2024-04-15T15:16:25.904156+0000","last_fullsized":"2024-04-15T15:16:25.904156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884989+0000","last_change":"2024-04-15T15:16:25.884088+0000","last_active":"2024-04-15T15:16:26.884989+0000","last_peered":"2024-04-15T15:16:26.884989+0000","last_clean":"2024-04-15T15:16:26.884989+0000","last_became_active":"2024-04-15T15:16:25.883804+0000","last_became_peered":"2024-04-15T15:16:25.883804+0000","last_unstale":"2024-04-15T15:16:26.884989+0000","last_undegraded":"2024-04-15T15:16:26.884989+0000","last_fullsized":"2024-04-15T15:16:26.884989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885622+0000","last_change":"2024-04-15T15:16:25.879663+0000","last_active":"2024-04-15T15:16:26.885622+0000","last_peered":"2024-04-15T15:16:26.885622+0000","last_clean":"2024-04-15T15:16:26.885622+0000","last_became_active":"2024-04-15T15:16:25.879385+0000","last_became_peered":"2024-04-15T15:16:25.879385+0000","last_unstale":"2024-04-15T15:16:26.885622+0000","last_undegraded":"2024-04-15T15:16:26.885622+0000","last_fullsized":"2024-04-15T15:16:26.885622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888033+0000","last_change":"2024-04-15T15:16:25.886972+0000","last_active":"2024-04-15T15:16:25.888033+0000","last_peered":"2024-04-15T15:16:25.888033+0000","last_clean":"2024-04-15T15:16:25.888033+0000","last_became_active":"2024-04-15T15:16:25.886718+0000","last_became_peered":"2024-04-15T15:16:25.886718+0000","last_unstale":"2024-04-15T15:16:25.888033+0000","last_undegraded":"2024-04-15T15:16:25.888033+0000","last_fullsized":"2024-04-15T15:16:25.888033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885577+0000","last_change":"2024-04-15T15:16:24.886178+0000","last_active":"2024-04-15T15:16:26.885577+0000","last_peered":"2024-04-15T15:16:26.885577+0000","last_clean":"2024-04-15T15:16:26.885577+0000","last_became_active":"2024-04-15T15:16:24.885910+0000","last_became_peered":"2024-04-15T15:16:24.885910+0000","last_unstale":"2024-04-15T15:16:26.885577+0000","last_undegraded":"2024-04-15T15:16:26.885577+0000","last_fullsized":"2024-04-15T15:16:26.885577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903528+0000","last_change":"2024-04-15T15:16:25.902820+0000","last_active":"2024-04-15T15:16:25.903528+0000","last_peered":"2024-04-15T15:16:25.903528+0000","last_clean":"2024-04-15T15:16:25.903528+0000","last_became_active":"2024-04-15T15:16:25.902552+0000","last_became_peered":"2024-04-15T15:16:25.902552+0000","last_unstale":"2024-04-15T15:16:25.903528+0000","last_undegraded":"2024-04-15T15:16:25.903528+0000","last_fullsized":"2024-04-15T15:16:25.903528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898338+0000","last_change":"2024-04-15T15:16:24.886331+0000","last_active":"2024-04-15T15:16:25.898338+0000","last_peered":"2024-04-15T15:16:25.898338+0000","last_clean":"2024-04-15T15:16:25.898338+0000","last_became_active":"2024-04-15T15:16:24.886052+0000","last_became_peered":"2024-04-15T15:16:24.886052+0000","last_unstale":"2024-04-15T15:16:25.898338+0000","last_undegraded":"2024-04-15T15:16:25.898338+0000","last_fullsized":"2024-04-15T15:16:25.898338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.899883+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.899620+0000","last_became_peered":"2024-04-15T15:16:25.899620+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898694+0000","last_change":"2024-04-15T15:16:24.890646+0000","last_active":"2024-04-15T15:16:25.898694+0000","last_peered":"2024-04-15T15:16:25.898694+0000","last_clean":"2024-04-15T15:16:25.898694+0000","last_became_active":"2024-04-15T15:16:24.890362+0000","last_became_peered":"2024-04-15T15:16:24.890362+0000","last_unstale":"2024-04-15T15:16:25.898694+0000","last_undegraded":"2024-04-15T15:16:25.898694+0000","last_fullsized":"2024-04-15T15:16:25.898694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883847+0000","last_change":"2024-04-15T15:16:25.895423+0000","last_active":"2024-04-15T15:16:26.883847+0000","last_peered":"2024-04-15T15:16:26.883847+0000","last_clean":"2024-04-15T15:16:26.883847+0000","last_became_active":"2024-04-15T15:16:25.889905+0000","last_became_peered":"2024-04-15T15:16:25.889905+0000","last_unstale":"2024-04-15T15:16:26.883847+0000","last_undegraded":"2024-04-15T15:16:26.883847+0000","last_fullsized":"2024-04-15T15:16:26.883847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:50:15.175711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886111+0000","last_change":"2024-04-15T15:16:24.887352+0000","last_active":"2024-04-15T15:16:26.886111+0000","last_peered":"2024-04-15T15:16:26.886111+0000","last_clean":"2024-04-15T15:16:26.886111+0000","last_became_active":"2024-04-15T15:16:24.887054+0000","last_became_peered":"2024-04-15T15:16:24.887054+0000","last_unstale":"2024-04-15T15:16:26.886111+0000","last_undegraded":"2024-04-15T15:16:26.886111+0000","last_fullsized":"2024-04-15T15:16:26.886111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.882206+0000","last_change":"2024-04-15T15:16:25.881247+0000","last_active":"2024-04-15T15:16:25.882206+0000","last_peered":"2024-04-15T15:16:25.882206+0000","last_clean":"2024-04-15T15:16:25.882206+0000","last_became_active":"2024-04-15T15:16:25.880981+0000","last_became_peered":"2024-04-15T15:16:25.880981+0000","last_unstale":"2024-04-15T15:16:25.882206+0000","last_undegraded":"2024-04-15T15:16:25.882206+0000","last_fullsized":"2024-04-15T15:16:25.882206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903854+0000","last_change":"2024-04-15T15:16:24.891022+0000","last_active":"2024-04-15T15:16:25.903854+0000","last_peered":"2024-04-15T15:16:25.903854+0000","last_clean":"2024-04-15T15:16:25.903854+0000","last_became_active":"2024-04-15T15:16:24.890770+0000","last_became_peered":"2024-04-15T15:16:24.890770+0000","last_unstale":"2024-04-15T15:16:25.903854+0000","last_undegraded":"2024-04-15T15:16:25.903854+0000","last_fullsized":"2024-04-15T15:16:25.903854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901334+0000","last_change":"2024-04-15T15:16:25.893573+0000","last_active":"2024-04-15T15:16:25.901334+0000","last_peered":"2024-04-15T15:16:25.901334+0000","last_clean":"2024-04-15T15:16:25.901334+0000","last_became_active":"2024-04-15T15:16:25.893321+0000","last_became_peered":"2024-04-15T15:16:25.893321+0000","last_unstale":"2024-04-15T15:16:25.901334+0000","last_undegraded":"2024-04-15T15:16:25.901334+0000","last_fullsized":"2024-04-15T15:16:25.901334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885832+0000","last_change":"2024-04-15T15:16:25.904897+0000","last_active":"2024-04-15T15:16:26.885832+0000","last_peered":"2024-04-15T15:16:26.885832+0000","last_clean":"2024-04-15T15:16:26.885832+0000","last_became_active":"2024-04-15T15:16:25.904610+0000","last_became_peered":"2024-04-15T15:16:25.904610+0000","last_unstale":"2024-04-15T15:16:26.885832+0000","last_undegraded":"2024-04-15T15:16:26.885832+0000","last_fullsized":"2024-04-15T15:16:26.885832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876235+0000","last_change":"2024-04-15T15:16:24.891428+0000","last_active":"2024-04-15T15:16:25.876235+0000","last_peered":"2024-04-15T15:16:25.876235+0000","last_clean":"2024-04-15T15:16:25.876235+0000","last_became_active":"2024-04-15T15:16:24.891151+0000","last_became_peered":"2024-04-15T15:16:24.891151+0000","last_unstale":"2024-04-15T15:16:25.876235+0000","last_undegraded":"2024-04-15T15:16:25.876235+0000","last_fullsized":"2024-04-15T15:16:25.876235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:56:54.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883767+0000","last_change":"2024-04-15T15:16:25.898140+0000","last_active":"2024-04-15T15:16:26.883767+0000","last_peered":"2024-04-15T15:16:26.883767+0000","last_clean":"2024-04-15T15:16:26.883767+0000","last_became_active":"2024-04-15T15:16:25.897492+0000","last_became_peered":"2024-04-15T15:16:25.897492+0000","last_unstale":"2024-04-15T15:16:26.883767+0000","last_undegraded":"2024-04-15T15:16:26.883767+0000","last_fullsized":"2024-04-15T15:16:26.883767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884775+0000","last_change":"2024-04-15T15:16:24.884571+0000","last_active":"2024-04-15T15:16:26.884775+0000","last_peered":"2024-04-15T15:16:26.884775+0000","last_clean":"2024-04-15T15:16:26.884775+0000","last_became_active":"2024-04-15T15:16:24.884274+0000","last_became_peered":"2024-04-15T15:16:24.884274+0000","last_unstale":"2024-04-15T15:16:26.884775+0000","last_undegraded":"2024-04-15T15:16:26.884775+0000","last_fullsized":"2024-04-15T15:16:26.884775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.899098+0000","last_change":"2024-04-15T15:16:25.898346+0000","last_active":"2024-04-15T15:16:25.899098+0000","last_peered":"2024-04-15T15:16:25.899098+0000","last_clean":"2024-04-15T15:16:25.899098+0000","last_became_active":"2024-04-15T15:16:25.897145+0000","last_became_peered":"2024-04-15T15:16:25.897145+0000","last_unstale":"2024-04-15T15:16:25.899098+0000","last_undegraded":"2024-04-15T15:16:25.899098+0000","last_fullsized":"2024-04-15T15:16:25.899098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:54.775491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884135+0000","last_change":"2024-04-15T15:16:24.888936+0000","last_active":"2024-04-15T15:16:26.884135+0000","last_peered":"2024-04-15T15:16:26.884135+0000","last_clean":"2024-04-15T15:16:26.884135+0000","last_became_active":"2024-04-15T15:16:24.888050+0000","last_became_peered":"2024-04-15T15:16:24.888050+0000","last_unstale":"2024-04-15T15:16:26.884135+0000","last_undegraded":"2024-04-15T15:16:26.884135+0000","last_fullsized":"2024-04-15T15:16:26.884135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.886799+0000","last_change":"2024-04-15T15:16:24.887199+0000","last_active":"2024-04-15T15:16:25.886799+0000","last_peered":"2024-04-15T15:16:25.886799+0000","last_clean":"2024-04-15T15:16:25.886799+0000","last_became_active":"2024-04-15T15:16:24.886945+0000","last_became_peered":"2024-04-15T15:16:24.886945+0000","last_unstale":"2024-04-15T15:16:25.886799+0000","last_undegraded":"2024-04-15T15:16:25.886799+0000","last_fullsized":"2024-04-15T15:16:25.886799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.893867+0000","last_change":"2024-04-15T15:16:25.892203+0000","last_active":"2024-04-15T15:16:25.893867+0000","last_peered":"2024-04-15T15:16:25.893867+0000","last_clean":"2024-04-15T15:16:25.893867+0000","last_became_active":"2024-04-15T15:16:25.891899+0000","last_became_peered":"2024-04-15T15:16:25.891899+0000","last_unstale":"2024-04-15T15:16:25.893867+0000","last_undegraded":"2024-04-15T15:16:25.893867+0000","last_fullsized":"2024-04-15T15:16:25.893867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904566+0000","last_change":"2024-04-15T15:16:24.890265+0000","last_active":"2024-04-15T15:16:25.904566+0000","last_peered":"2024-04-15T15:16:25.904566+0000","last_clean":"2024-04-15T15:16:25.904566+0000","last_became_active":"2024-04-15T15:16:24.889963+0000","last_became_peered":"2024-04-15T15:16:24.889963+0000","last_unstale":"2024-04-15T15:16:25.904566+0000","last_undegraded":"2024-04-15T15:16:25.904566+0000","last_fullsized":"2024-04-15T15:16:25.904566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886503+0000","last_change":"2024-04-15T15:16:25.893921+0000","last_active":"2024-04-15T15:16:26.886503+0000","last_peered":"2024-04-15T15:16:26.886503+0000","last_clean":"2024-04-15T15:16:26.886503+0000","last_became_active":"2024-04-15T15:16:25.892154+0000","last_became_peered":"2024-04-15T15:16:25.892154+0000","last_unstale":"2024-04-15T15:16:26.886503+0000","last_undegraded":"2024-04-15T15:16:26.886503+0000","last_fullsized":"2024-04-15T15:16:26.886503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884185+0000","last_change":"2024-04-15T15:16:24.889313+0000","last_active":"2024-04-15T15:16:26.884185+0000","last_peered":"2024-04-15T15:16:26.884185+0000","last_clean":"2024-04-15T15:16:26.884185+0000","last_became_active":"2024-04-15T15:16:24.889037+0000","last_became_peered":"2024-04-15T15:16:24.889037+0000","last_unstale":"2024-04-15T15:16:26.884185+0000","last_undegraded":"2024-04-15T15:16:26.884185+0000","last_fullsized":"2024-04-15T15:16:26.884185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:58.773260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.889667+0000","last_change":"2024-04-15T15:16:25.888305+0000","last_active":"2024-04-15T15:16:25.889667+0000","last_peered":"2024-04-15T15:16:25.889667+0000","last_clean":"2024-04-15T15:16:25.889667+0000","last_became_active":"2024-04-15T15:16:25.888053+0000","last_became_peered":"2024-04-15T15:16:25.888053+0000","last_unstale":"2024-04-15T15:16:25.889667+0000","last_undegraded":"2024-04-15T15:16:25.889667+0000","last_fullsized":"2024-04-15T15:16:25.889667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884136+0000","last_change":"2024-04-15T15:16:24.882824+0000","last_active":"2024-04-15T15:16:26.884136+0000","last_peered":"2024-04-15T15:16:26.884136+0000","last_clean":"2024-04-15T15:16:26.884136+0000","last_became_active":"2024-04-15T15:16:24.882560+0000","last_became_peered":"2024-04-15T15:16:24.882560+0000","last_unstale":"2024-04-15T15:16:26.884136+0000","last_undegraded":"2024-04-15T15:16:26.884136+0000","last_fullsized":"2024-04-15T15:16:26.884136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:08.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883781+0000","last_change":"2024-04-15T15:16:25.883512+0000","last_active":"2024-04-15T15:16:26.883781+0000","last_peered":"2024-04-15T15:16:26.883781+0000","last_clean":"2024-04-15T15:16:26.883781+0000","last_became_active":"2024-04-15T15:16:25.883253+0000","last_became_peered":"2024-04-15T15:16:25.883253+0000","last_unstale":"2024-04-15T15:16:26.883781+0000","last_undegraded":"2024-04-15T15:16:26.883781+0000","last_fullsized":"2024-04-15T15:16:26.883781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.877572+0000","last_change":"2024-04-15T15:16:24.891194+0000","last_active":"2024-04-15T15:16:25.877572+0000","last_peered":"2024-04-15T15:16:25.877572+0000","last_clean":"2024-04-15T15:16:25.877572+0000","last_became_active":"2024-04-15T15:16:24.890944+0000","last_became_peered":"2024-04-15T15:16:24.890944+0000","last_unstale":"2024-04-15T15:16:25.877572+0000","last_undegraded":"2024-04-15T15:16:25.877572+0000","last_fullsized":"2024-04-15T15:16:25.877572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885147+0000","last_change":"2024-04-15T15:16:24.886029+0000","last_active":"2024-04-15T15:16:26.885147+0000","last_peered":"2024-04-15T15:16:26.885147+0000","last_clean":"2024-04-15T15:16:26.885147+0000","last_became_active":"2024-04-15T15:16:24.885730+0000","last_became_peered":"2024-04-15T15:16:24.885730+0000","last_unstale":"2024-04-15T15:16:26.885147+0000","last_undegraded":"2024-04-15T15:16:26.885147+0000","last_fullsized":"2024-04-15T15:16:26.885147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884020+0000","last_change":"2024-04-15T15:16:25.882916+0000","last_active":"2024-04-15T15:16:26.884020+0000","last_peered":"2024-04-15T15:16:26.884020+0000","last_clean":"2024-04-15T15:16:26.884020+0000","last_became_active":"2024-04-15T15:16:25.882630+0000","last_became_peered":"2024-04-15T15:16:25.882630+0000","last_unstale":"2024-04-15T15:16:26.884020+0000","last_undegraded":"2024-04-15T15:16:26.884020+0000","last_fullsized":"2024-04-15T15:16:26.884020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:24.310942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884117+0000","last_change":"2024-04-15T15:16:25.890929+0000","last_active":"2024-04-15T15:16:26.884117+0000","last_peered":"2024-04-15T15:16:26.884117+0000","last_clean":"2024-04-15T15:16:26.884117+0000","last_became_active":"2024-04-15T15:16:25.890256+0000","last_became_peered":"2024-04-15T15:16:25.890256+0000","last_unstale":"2024-04-15T15:16:26.884117+0000","last_undegraded":"2024-04-15T15:16:26.884117+0000","last_fullsized":"2024-04-15T15:16:26.884117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:11.834293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906758+0000","last_change":"2024-04-15T15:16:24.893507+0000","last_active":"2024-04-15T15:16:25.906758+0000","last_peered":"2024-04-15T15:16:25.906758+0000","last_clean":"2024-04-15T15:16:25.906758+0000","last_became_active":"2024-04-15T15:16:24.893262+0000","last_became_peered":"2024-04-15T15:16:24.893262+0000","last_unstale":"2024-04-15T15:16:25.906758+0000","last_undegraded":"2024-04-15T15:16:25.906758+0000","last_fullsized":"2024-04-15T15:16:25.906758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:15.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911705+0000","last_change":"2024-04-15T15:16:25.904384+0000","last_active":"2024-04-15T15:16:25.911705+0000","last_peered":"2024-04-15T15:16:25.911705+0000","last_clean":"2024-04-15T15:16:25.911705+0000","last_became_active":"2024-04-15T15:16:25.904095+0000","last_became_peered":"2024-04-15T15:16:25.904095+0000","last_unstale":"2024-04-15T15:16:25.911705+0000","last_undegraded":"2024-04-15T15:16:25.911705+0000","last_fullsized":"2024-04-15T15:16:25.911705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886583+0000","last_change":"2024-04-15T15:16:24.891423+0000","last_active":"2024-04-15T15:16:26.886583+0000","last_peered":"2024-04-15T15:16:26.886583+0000","last_clean":"2024-04-15T15:16:26.886583+0000","last_became_active":"2024-04-15T15:16:24.891048+0000","last_became_peered":"2024-04-15T15:16:24.891048+0000","last_unstale":"2024-04-15T15:16:26.886583+0000","last_undegraded":"2024-04-15T15:16:26.886583+0000","last_fullsized":"2024-04-15T15:16:26.886583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:47:10.367180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.894040+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.893781+0000","last_became_peered":"2024-04-15T15:16:25.893781+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.897493+0000","last_change":"2024-04-15T15:16:24.889598+0000","last_active":"2024-04-15T15:16:25.897493+0000","last_peered":"2024-04-15T15:16:25.897493+0000","last_clean":"2024-04-15T15:16:25.897493+0000","last_became_active":"2024-04-15T15:16:24.889298+0000","last_became_peered":"2024-04-15T15:16:24.889298+0000","last_unstale":"2024-04-15T15:16:25.897493+0000","last_undegraded":"2024-04-15T15:16:25.897493+0000","last_fullsized":"2024-04-15T15:16:25.897493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:54:18.385006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891485+0000","last_change":"2024-04-15T15:16:25.884522+0000","last_active":"2024-04-15T15:16:25.891485+0000","last_peered":"2024-04-15T15:16:25.891485+0000","last_clean":"2024-04-15T15:16:25.891485+0000","last_became_active":"2024-04-15T15:16:25.883126+0000","last_became_peered":"2024-04-15T15:16:25.883126+0000","last_unstale":"2024-04-15T15:16:25.891485+0000","last_undegraded":"2024-04-15T15:16:25.891485+0000","last_fullsized":"2024-04-15T15:16:25.891485+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:03.969456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876163+0000","last_change":"2024-04-15T15:16:24.890031+0000","last_active":"2024-04-15T15:16:25.876163+0000","last_peered":"2024-04-15T15:16:25.876163+0000","last_clean":"2024-04-15T15:16:25.876163+0000","last_became_active":"2024-04-15T15:16:24.889203+0000","last_became_peered":"2024-04-15T15:16:24.889203+0000","last_unstale":"2024-04-15T15:16:25.876163+0000","last_undegraded":"2024-04-15T15:16:25.876163+0000","last_fullsized":"2024-04-15T15:16:25.876163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:18.919033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891401+0000","last_change":"2024-04-15T15:16:24.877420+0000","last_active":"2024-04-15T15:16:25.891401+0000","last_peered":"2024-04-15T15:16:25.891401+0000","last_clean":"2024-04-15T15:16:25.891401+0000","last_became_active":"2024-04-15T15:16:24.877149+0000","last_became_peered":"2024-04-15T15:16:24.877149+0000","last_unstale":"2024-04-15T15:16:25.891401+0000","last_undegraded":"2024-04-15T15:16:25.891401+0000","last_fullsized":"2024-04-15T15:16:25.891401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:48.237807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.890064+0000","last_change":"2024-04-15T15:16:25.887625+0000","last_active":"2024-04-15T15:16:25.890064+0000","last_peered":"2024-04-15T15:16:25.890064+0000","last_clean":"2024-04-15T15:16:25.890064+0000","last_became_active":"2024-04-15T15:16:25.887362+0000","last_became_peered":"2024-04-15T15:16:25.887362+0000","last_unstale":"2024-04-15T15:16:25.890064+0000","last_undegraded":"2024-04-15T15:16:25.890064+0000","last_fullsized":"2024-04-15T15:16:25.890064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872610+0000","last_change":"2024-04-15T15:16:24.884150+0000","last_active":"2024-04-15T15:16:25.872610+0000","last_peered":"2024-04-15T15:16:25.872610+0000","last_clean":"2024-04-15T15:16:25.872610+0000","last_became_active":"2024-04-15T15:16:24.883908+0000","last_became_peered":"2024-04-15T15:16:24.883908+0000","last_unstale":"2024-04-15T15:16:25.872610+0000","last_undegraded":"2024-04-15T15:16:25.872610+0000","last_fullsized":"2024-04-15T15:16:25.872610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:58:35.147306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903908+0000","last_change":"2024-04-15T15:16:25.897701+0000","last_active":"2024-04-15T15:16:25.903908+0000","last_peered":"2024-04-15T15:16:25.903908+0000","last_clean":"2024-04-15T15:16:25.903908+0000","last_became_active":"2024-04-15T15:16:25.896891+0000","last_became_peered":"2024-04-15T15:16:25.896891+0000","last_unstale":"2024-04-15T15:16:25.903908+0000","last_undegraded":"2024-04-15T15:16:25.903908+0000","last_fullsized":"2024-04-15T15:16:25.903908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884030+0000","last_change":"2024-04-15T15:16:25.899465+0000","last_active":"2024-04-15T15:16:26.884030+0000","last_peered":"2024-04-15T15:16:26.884030+0000","last_clean":"2024-04-15T15:16:26.884030+0000","last_became_active":"2024-04-15T15:16:25.899205+0000","last_became_peered":"2024-04-15T15:16:25.899205+0000","last_unstale":"2024-04-15T15:16:26.884030+0000","last_undegraded":"2024-04-15T15:16:26.884030+0000","last_fullsized":"2024-04-15T15:16:26.884030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:42:16.777997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872575+0000","last_change":"2024-04-15T15:16:24.882714+0000","last_active":"2024-04-15T15:16:25.872575+0000","last_peered":"2024-04-15T15:16:25.872575+0000","last_clean":"2024-04-15T15:16:25.872575+0000","last_became_active":"2024-04-15T15:16:24.882460+0000","last_became_peered":"2024-04-15T15:16:24.882460+0000","last_unstale":"2024-04-15T15:16:25.872575+0000","last_undegraded":"2024-04-15T15:16:25.872575+0000","last_fullsized":"2024-04-15T15:16:25.872575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:11.251738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898832+0000","last_change":"2024-04-15T15:16:25.897148+0000","last_active":"2024-04-15T15:16:25.898832+0000","last_peered":"2024-04-15T15:16:25.898832+0000","last_clean":"2024-04-15T15:16:25.898832+0000","last_became_active":"2024-04-15T15:16:25.896866+0000","last_became_peered":"2024-04-15T15:16:25.896866+0000","last_unstale":"2024-04-15T15:16:25.898832+0000","last_undegraded":"2024-04-15T15:16:25.898832+0000","last_fullsized":"2024-04-15T15:16:25.898832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902029+0000","last_change":"2024-04-15T15:16:24.892956+0000","last_active":"2024-04-15T15:16:25.902029+0000","last_peered":"2024-04-15T15:16:25.902029+0000","last_clean":"2024-04-15T15:16:25.902029+0000","last_became_active":"2024-04-15T15:16:24.892664+0000","last_became_peered":"2024-04-15T15:16:24.892664+0000","last_unstale":"2024-04-15T15:16:25.902029+0000","last_undegraded":"2024-04-15T15:16:25.902029+0000","last_fullsized":"2024-04-15T15:16:25.902029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:56.013415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.873000+0000","last_change":"2024-04-15T15:16:24.890594+0000","last_active":"2024-04-15T15:16:25.873000+0000","last_peered":"2024-04-15T15:16:25.873000+0000","last_clean":"2024-04-15T15:16:25.873000+0000","last_became_active":"2024-04-15T15:16:24.890305+0000","last_became_peered":"2024-04-15T15:16:24.890305+0000","last_unstale":"2024-04-15T15:16:25.873000+0000","last_undegraded":"2024-04-15T15:16:25.873000+0000","last_fullsized":"2024-04-15T15:16:25.873000+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:34.555529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884060+0000","last_change":"2024-04-15T15:16:25.893319+0000","last_active":"2024-04-15T15:16:26.884060+0000","last_peered":"2024-04-15T15:16:26.884060+0000","last_clean":"2024-04-15T15:16:26.884060+0000","last_became_active":"2024-04-15T15:16:25.892794+0000","last_became_peered":"2024-04-15T15:16:25.892794+0000","last_unstale":"2024-04-15T15:16:26.884060+0000","last_undegraded":"2024-04-15T15:16:26.884060+0000","last_fullsized":"2024-04-15T15:16:26.884060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:42.664115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876776+0000","last_change":"2024-04-15T15:16:24.884802+0000","last_active":"2024-04-15T15:16:25.876776+0000","last_peered":"2024-04-15T15:16:25.876776+0000","last_clean":"2024-04-15T15:16:25.876776+0000","last_became_active":"2024-04-15T15:16:24.884519+0000","last_became_peered":"2024-04-15T15:16:24.884519+0000","last_unstale":"2024-04-15T15:16:25.876776+0000","last_undegraded":"2024-04-15T15:16:25.876776+0000","last_fullsized":"2024-04-15T15:16:25.876776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888791+0000","last_change":"2024-04-15T15:16:25.883444+0000","last_active":"2024-04-15T15:16:25.888791+0000","last_peered":"2024-04-15T15:16:25.888791+0000","last_clean":"2024-04-15T15:16:25.888791+0000","last_became_active":"2024-04-15T15:16:25.883191+0000","last_became_peered":"2024-04-15T15:16:25.883191+0000","last_unstale":"2024-04-15T15:16:25.888791+0000","last_undegraded":"2024-04-15T15:16:25.888791+0000","last_fullsized":"2024-04-15T15:16:25.888791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.875590+0000","last_change":"2024-04-15T15:16:24.886148+0000","last_active":"2024-04-15T15:16:25.875590+0000","last_peered":"2024-04-15T15:16:25.875590+0000","last_clean":"2024-04-15T15:16:25.875590+0000","last_became_active":"2024-04-15T15:16:24.885865+0000","last_became_peered":"2024-04-15T15:16:24.885865+0000","last_unstale":"2024-04-15T15:16:25.875590+0000","last_undegraded":"2024-04-15T15:16:25.875590+0000","last_fullsized":"2024-04-15T15:16:25.875590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888002+0000","last_change":"2024-04-15T15:16:25.886178+0000","last_active":"2024-04-15T15:16:25.888002+0000","last_peered":"2024-04-15T15:16:25.888002+0000","last_clean":"2024-04-15T15:16:25.888002+0000","last_became_active":"2024-04-15T15:16:25.885915+0000","last_became_peered":"2024-04-15T15:16:25.885915+0000","last_unstale":"2024-04-15T15:16:25.888002+0000","last_undegraded":"2024-04-15T15:16:25.888002+0000","last_fullsized":"2024-04-15T15:16:25.888002+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:33.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885719+0000","last_change":"2024-04-15T15:16:24.883958+0000","last_active":"2024-04-15T15:16:26.885719+0000","last_peered":"2024-04-15T15:16:26.885719+0000","last_clean":"2024-04-15T15:16:26.885719+0000","last_became_active":"2024-04-15T15:16:24.883528+0000","last_became_peered":"2024-04-15T15:16:24.883528+0000","last_unstale":"2024-04-15T15:16:26.885719+0000","last_undegraded":"2024-04-15T15:16:26.885719+0000","last_fullsized":"2024-04-15T15:16:26.885719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885492+0000","last_change":"2024-04-15T15:16:25.889861+0000","last_active":"2024-04-15T15:16:26.885492+0000","last_peered":"2024-04-15T15:16:26.885492+0000","last_clean":"2024-04-15T15:16:26.885492+0000","last_became_active":"2024-04-15T15:16:25.889591+0000","last_became_peered":"2024-04-15T15:16:25.889591+0000","last_unstale":"2024-04-15T15:16:26.885492+0000","last_undegraded":"2024-04-15T15:16:26.885492+0000","last_fullsized":"2024-04-15T15:16:26.885492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884867+0000","last_change":"2024-04-15T15:16:24.891429+0000","last_active":"2024-04-15T15:16:26.884867+0000","last_peered":"2024-04-15T15:16:26.884867+0000","last_clean":"2024-04-15T15:16:26.884867+0000","last_became_active":"2024-04-15T15:16:24.891111+0000","last_became_peered":"2024-04-15T15:16:24.891111+0000","last_unstale":"2024-04-15T15:16:26.884867+0000","last_undegraded":"2024-04-15T15:16:26.884867+0000","last_fullsized":"2024-04-15T15:16:26.884867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883631+0000","last_change":"2024-04-15T15:16:24.887452+0000","last_active":"2024-04-15T15:16:26.883631+0000","last_peered":"2024-04-15T15:16:26.883631+0000","last_clean":"2024-04-15T15:16:26.883631+0000","last_became_active":"2024-04-15T15:16:24.887136+0000","last_became_peered":"2024-04-15T15:16:24.887136+0000","last_unstale":"2024-04-15T15:16:26.883631+0000","last_undegraded":"2024-04-15T15:16:26.883631+0000","last_fullsized":"2024-04-15T15:16:26.883631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:24.817169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906655+0000","last_change":"2024-04-15T15:16:25.901054+0000","last_active":"2024-04-15T15:16:25.906655+0000","last_peered":"2024-04-15T15:16:25.906655+0000","last_clean":"2024-04-15T15:16:25.906655+0000","last_became_active":"2024-04-15T15:16:25.900784+0000","last_became_peered":"2024-04-15T15:16:25.900784+0000","last_unstale":"2024-04-15T15:16:25.906655+0000","last_undegraded":"2024-04-15T15:16:25.906655+0000","last_fullsized":"2024-04-15T15:16:25.906655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-15T15:16:20.700430+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:16:20.700430+0000","last_peered":"2024-04-15T15:16:20.700430+0000","last_clean":"2024-04-15T15:16:20.700430+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:16:20.700430+0000","last_undegraded":"2024-04-15T15:16:20.700430+0000","last_fullsized":"2024-04-15T15:16:20.700430+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":108,"acting":108,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":56,"acting":56,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":758052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":189373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":778129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:28.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:29.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.023+0000 7f6334fdb640 1 Processor -- start 2024-04-15T15:16:29.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.023+0000 7f6334fdb640 1 -- start start 2024-04-15T15:16:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f6334fdb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 0x7f63300d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f6334fdb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f63300b3160 0x7f633015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f6334fdb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63300d0fb0 con 0x7f633015ef70 2024-04-15T15:16:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f6334fdb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f633015cb50 con 0x7f63300b3160 2024-04-15T15:16:29.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f6334fdb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f633015bfd0 con 0x7f63300d0690 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f632dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f63300b3160 0x7f633015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f632dd9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f63300b3160 0x7f633015b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49780/0 (socket says 172.21.15.70:49780) 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.024+0000 7f632dd9b640 1 -- 172.21.15.70:0/2946686049 learned_addr learned my addr 172.21.15.70:0/2946686049 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632e59c640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 0x7f63300d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632dd9b640 1 -- 172.21.15.70:0/2946686049 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 msgr2=0x7f63300d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632dd9b640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 0x7f63300d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632dd9b640 1 -- 172.21.15.70:0/2946686049 >> v1:172.21.15.96:6789/0 conn(0x7f633015ef70 legacy=0x7f633015f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632dd9b640 1 -- 172.21.15.70:0/2946686049 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f633015c2b0 con 0x7f63300b3160 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632e59c640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 0x7f63300d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632dd9b640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300b3160 0x7f633015b690 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6324009870 tx=0x7f632402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445d72ec94d38a96 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.025+0000 7f632d59a640 1 -- 172.21.15.70:0/2946686049 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324035020 con 0x7f63300b3160 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f632d59a640 1 -- 172.21.15.70:0/2946686049 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6324031640 con 0x7f63300b3160 2024-04-15T15:16:29.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f632d59a640 1 -- 172.21.15.70:0/2946686049 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632402d040 con 0x7f63300b3160 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 -- 172.21.15.70:0/2946686049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300b3160 msgr2=0x7f633015b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300b3160 0x7f633015b690 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f6324009870 tx=0x7f632402b8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 -- 172.21.15.70:0/2946686049 shutdown_connections 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f63300b3160 0x7f633015b690 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 --2- 172.21.15.70:0/2946686049 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300d0690 0x7f63300d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 -- 172.21.15.70:0/2946686049 >> 172.21.15.70:0/2946686049 conn(0x7f633006be00 msgr2=0x7f633006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 -- 172.21.15.70:0/2946686049 shutdown_connections 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.026+0000 7f6334fdb640 1 -- 172.21.15.70:0/2946686049 wait complete. 2024-04-15T15:16:29.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.027+0000 7f6334fdb640 1 Processor -- start 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 -- start start 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300b3160 0x7f63301e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63300d0690 0x7f6330153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f63300b4fb0 con 0x7f633015ef70 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f633015c870 con 0x7f63300d0690 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f6334fdb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f633015bbd0 con 0x7f63300b3160 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f632ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f632ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49784/0 (socket says 172.21.15.70:49784) 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.028+0000 7f632ed9d640 1 -- 172.21.15.70:0/419700987 learned_addr learned my addr 172.21.15.70:0/419700987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 -- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300b3160 msgr2=0x7f63301e4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300b3160 0x7f63301e4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 -- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63300d0690 msgr2=0x7f6330153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632dd9b640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63300d0690 0x7f6330153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63300d0690 0x7f6330153640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 -- 172.21.15.70:0/419700987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63300cc650 con 0x7f633015ef70 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f632ed9d640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f632800b410 tx=0x7f632800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328015030 con 0x7f633015ef70 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6328004020 con 0x7f633015ef70 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63301e5700 con 0x7f633015ef70 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328004410 con 0x7f633015ef70 2024-04-15T15:16:29.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.029+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63301e5cc0 con 0x7f633015ef70 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.030+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63300689e0 con 0x7f633015ef70 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.030+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f6328018070 con 0x7f633015ef70 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.031+0000 7f631effd640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 0x7f631006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.031+0000 7f632e59c640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 0x7f631006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.031+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f632808b790 con 0x7f633015ef70 2024-04-15T15:16:29.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.031+0000 7f632e59c640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 0x7f631006edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6320008e00 tx=0x7f6320008330 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.035+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6328059a30 con 0x7f633015ef70 2024-04-15T15:16:29.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.207+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f63300739e0 con 0x7f631006c8f0 2024-04-15T15:16:29.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.213+0000 7f631effd640 1 -- 172.21.15.70:0/419700987 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249848 (secure 0 0 0) 0x7f63300739e0 con 0x7f631006c8f0 2024-04-15T15:16:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:29.218 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 msgr2=0x7f631006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 0x7f631006edb0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6320008e00 tx=0x7f6320008330 comp rx=0 tx=0).stop 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 msgr2=0x7f63300cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f632800b410 tx=0x7f632800b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 shutdown_connections 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f631006c8f0 0x7f631006edb0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.215+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f633015ef70 0x7f63300cb4c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.216+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f63300d0690 0x7f6330153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.216+0000 7f6334fdb640 1 --2- 172.21.15.70:0/419700987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f63300b3160 0x7f63301e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.216+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 >> 172.21.15.70:0/419700987 conn(0x7f633006be00 msgr2=0x7f633015f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.216+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 shutdown_connections 2024-04-15T15:16:29.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.216+0000 7f6334fdb640 1 -- 172.21.15.70:0/419700987 wait complete. 2024-04-15T15:16:29.228 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-15T15:16:28.085226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":191,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":166,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237408,"kb_used_data":4596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4706304,"data_stored":2665647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.219711"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884133+0000","last_change":"2024-04-15T15:16:25.883054+0000","last_active":"2024-04-15T15:16:26.884133+0000","last_peered":"2024-04-15T15:16:26.884133+0000","last_clean":"2024-04-15T15:16:26.884133+0000","last_became_active":"2024-04-15T15:16:25.882744+0000","last_became_peered":"2024-04-15T15:16:25.882744+0000","last_unstale":"2024-04-15T15:16:26.884133+0000","last_undegraded":"2024-04-15T15:16:26.884133+0000","last_fullsized":"2024-04-15T15:16:26.884133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:06:55.204459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883810+0000","last_change":"2024-04-15T15:16:25.888729+0000","last_active":"2024-04-15T15:16:26.883810+0000","last_peered":"2024-04-15T15:16:26.883810+0000","last_clean":"2024-04-15T15:16:26.883810+0000","last_became_active":"2024-04-15T15:16:25.888450+0000","last_became_peered":"2024-04-15T15:16:25.888450+0000","last_unstale":"2024-04-15T15:16:26.883810+0000","last_undegraded":"2024-04-15T15:16:26.883810+0000","last_fullsized":"2024-04-15T15:16:26.883810+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902977+0000","last_change":"2024-04-15T15:16:25.902269+0000","last_active":"2024-04-15T15:16:25.902977+0000","last_peered":"2024-04-15T15:16:25.902977+0000","last_clean":"2024-04-15T15:16:25.902977+0000","last_became_active":"2024-04-15T15:16:25.901972+0000","last_became_peered":"2024-04-15T15:16:25.901972+0000","last_unstale":"2024-04-15T15:16:25.902977+0000","last_undegraded":"2024-04-15T15:16:25.902977+0000","last_fullsized":"2024-04-15T15:16:25.902977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898887+0000","last_change":"2024-04-15T15:16:25.897033+0000","last_active":"2024-04-15T15:16:25.898887+0000","last_peered":"2024-04-15T15:16:25.898887+0000","last_clean":"2024-04-15T15:16:25.898887+0000","last_became_active":"2024-04-15T15:16:25.896432+0000","last_became_peered":"2024-04-15T15:16:25.896432+0000","last_unstale":"2024-04-15T15:16:25.898887+0000","last_undegraded":"2024-04-15T15:16:25.898887+0000","last_fullsized":"2024-04-15T15:16:25.898887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:19.565523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891565+0000","last_change":"2024-04-15T15:16:25.889069+0000","last_active":"2024-04-15T15:16:25.891565+0000","last_peered":"2024-04-15T15:16:25.891565+0000","last_clean":"2024-04-15T15:16:25.891565+0000","last_became_active":"2024-04-15T15:16:25.888793+0000","last_became_peered":"2024-04-15T15:16:25.888793+0000","last_unstale":"2024-04-15T15:16:25.891565+0000","last_undegraded":"2024-04-15T15:16:25.891565+0000","last_fullsized":"2024-04-15T15:16:25.891565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:48.611396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888862+0000","last_change":"2024-04-15T15:16:25.887072+0000","last_active":"2024-04-15T15:16:25.888862+0000","last_peered":"2024-04-15T15:16:25.888862+0000","last_clean":"2024-04-15T15:16:25.888862+0000","last_became_active":"2024-04-15T15:16:25.886821+0000","last_became_peered":"2024-04-15T15:16:25.886821+0000","last_unstale":"2024-04-15T15:16:25.888862+0000","last_undegraded":"2024-04-15T15:16:25.888862+0000","last_fullsized":"2024-04-15T15:16:25.888862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904107+0000","last_change":"2024-04-15T15:16:25.900378+0000","last_active":"2024-04-15T15:16:25.904107+0000","last_peered":"2024-04-15T15:16:25.904107+0000","last_clean":"2024-04-15T15:16:25.904107+0000","last_became_active":"2024-04-15T15:16:25.900071+0000","last_became_peered":"2024-04-15T15:16:25.900071+0000","last_unstale":"2024-04-15T15:16:25.904107+0000","last_undegraded":"2024-04-15T15:16:25.904107+0000","last_fullsized":"2024-04-15T15:16:25.904107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:43.288974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883618+0000","last_change":"2024-04-15T15:16:25.893507+0000","last_active":"2024-04-15T15:16:26.883618+0000","last_peered":"2024-04-15T15:16:26.883618+0000","last_clean":"2024-04-15T15:16:26.883618+0000","last_became_active":"2024-04-15T15:16:25.893005+0000","last_became_peered":"2024-04-15T15:16:25.893005+0000","last_unstale":"2024-04-15T15:16:26.883618+0000","last_undegraded":"2024-04-15T15:16:26.883618+0000","last_fullsized":"2024-04-15T15:16:26.883618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.895340+0000","last_change":"2024-04-15T15:16:25.894870+0000","last_active":"2024-04-15T15:16:25.895340+0000","last_peered":"2024-04-15T15:16:25.895340+0000","last_clean":"2024-04-15T15:16:25.895340+0000","last_became_active":"2024-04-15T15:16:25.894343+0000","last_became_peered":"2024-04-15T15:16:25.894343+0000","last_unstale":"2024-04-15T15:16:25.895340+0000","last_undegraded":"2024-04-15T15:16:25.895340+0000","last_fullsized":"2024-04-15T15:16:25.895340+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904248+0000","last_change":"2024-04-15T15:16:25.900279+0000","last_active":"2024-04-15T15:16:25.904248+0000","last_peered":"2024-04-15T15:16:25.904248+0000","last_clean":"2024-04-15T15:16:25.904248+0000","last_became_active":"2024-04-15T15:16:25.899969+0000","last_became_peered":"2024-04-15T15:16:25.899969+0000","last_unstale":"2024-04-15T15:16:25.904248+0000","last_undegraded":"2024-04-15T15:16:25.904248+0000","last_fullsized":"2024-04-15T15:16:25.904248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:19:11.196336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911723+0000","last_change":"2024-04-15T15:16:25.904761+0000","last_active":"2024-04-15T15:16:25.911723+0000","last_peered":"2024-04-15T15:16:25.911723+0000","last_clean":"2024-04-15T15:16:25.911723+0000","last_became_active":"2024-04-15T15:16:25.904477+0000","last_became_peered":"2024-04-15T15:16:25.904477+0000","last_unstale":"2024-04-15T15:16:25.911723+0000","last_undegraded":"2024-04-15T15:16:25.911723+0000","last_fullsized":"2024-04-15T15:16:25.911723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884502+0000","last_change":"2024-04-15T15:16:25.882646+0000","last_active":"2024-04-15T15:16:25.884502+0000","last_peered":"2024-04-15T15:16:25.884502+0000","last_clean":"2024-04-15T15:16:25.884502+0000","last_became_active":"2024-04-15T15:16:25.882375+0000","last_became_peered":"2024-04-15T15:16:25.882375+0000","last_unstale":"2024-04-15T15:16:25.884502+0000","last_undegraded":"2024-04-15T15:16:25.884502+0000","last_fullsized":"2024-04-15T15:16:25.884502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906634+0000","last_change":"2024-04-15T15:16:25.901003+0000","last_active":"2024-04-15T15:16:25.906634+0000","last_peered":"2024-04-15T15:16:25.906634+0000","last_clean":"2024-04-15T15:16:25.906634+0000","last_became_active":"2024-04-15T15:16:25.900729+0000","last_became_peered":"2024-04-15T15:16:25.900729+0000","last_unstale":"2024-04-15T15:16:25.906634+0000","last_undegraded":"2024-04-15T15:16:25.906634+0000","last_fullsized":"2024-04-15T15:16:25.906634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:34.769819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885365+0000","last_change":"2024-04-15T15:16:25.899260+0000","last_active":"2024-04-15T15:16:26.885365+0000","last_peered":"2024-04-15T15:16:26.885365+0000","last_clean":"2024-04-15T15:16:26.885365+0000","last_became_active":"2024-04-15T15:16:25.898979+0000","last_became_peered":"2024-04-15T15:16:25.898979+0000","last_unstale":"2024-04-15T15:16:26.885365+0000","last_undegraded":"2024-04-15T15:16:26.885365+0000","last_fullsized":"2024-04-15T15:16:26.885365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904064+0000","last_change":"2024-04-15T15:16:25.899744+0000","last_active":"2024-04-15T15:16:25.904064+0000","last_peered":"2024-04-15T15:16:25.904064+0000","last_clean":"2024-04-15T15:16:25.904064+0000","last_became_active":"2024-04-15T15:16:25.899472+0000","last_became_peered":"2024-04-15T15:16:25.899472+0000","last_unstale":"2024-04-15T15:16:25.904064+0000","last_undegraded":"2024-04-15T15:16:25.904064+0000","last_fullsized":"2024-04-15T15:16:25.904064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885281+0000","last_change":"2024-04-15T15:16:25.892123+0000","last_active":"2024-04-15T15:16:26.885281+0000","last_peered":"2024-04-15T15:16:26.885281+0000","last_clean":"2024-04-15T15:16:26.885281+0000","last_became_active":"2024-04-15T15:16:25.891853+0000","last_became_peered":"2024-04-15T15:16:25.891853+0000","last_unstale":"2024-04-15T15:16:26.885281+0000","last_undegraded":"2024-04-15T15:16:26.885281+0000","last_fullsized":"2024-04-15T15:16:26.885281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884688+0000","last_change":"2024-04-15T15:16:25.880039+0000","last_active":"2024-04-15T15:16:25.884688+0000","last_peered":"2024-04-15T15:16:25.884688+0000","last_clean":"2024-04-15T15:16:25.884688+0000","last_became_active":"2024-04-15T15:16:25.879753+0000","last_became_peered":"2024-04-15T15:16:25.879753+0000","last_unstale":"2024-04-15T15:16:25.884688+0000","last_undegraded":"2024-04-15T15:16:25.884688+0000","last_fullsized":"2024-04-15T15:16:25.884688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:58:15.988048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885269+0000","last_change":"2024-04-15T15:16:25.897291+0000","last_active":"2024-04-15T15:16:26.885269+0000","last_peered":"2024-04-15T15:16:26.885269+0000","last_clean":"2024-04-15T15:16:26.885269+0000","last_became_active":"2024-04-15T15:16:25.896977+0000","last_became_peered":"2024-04-15T15:16:25.896977+0000","last_unstale":"2024-04-15T15:16:26.885269+0000","last_undegraded":"2024-04-15T15:16:26.885269+0000","last_fullsized":"2024-04-15T15:16:26.885269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901153+0000","last_change":"2024-04-15T15:16:25.889177+0000","last_active":"2024-04-15T15:16:25.901153+0000","last_peered":"2024-04-15T15:16:25.901153+0000","last_clean":"2024-04-15T15:16:25.901153+0000","last_became_active":"2024-04-15T15:16:25.888843+0000","last_became_peered":"2024-04-15T15:16:25.888843+0000","last_unstale":"2024-04-15T15:16:25.901153+0000","last_undegraded":"2024-04-15T15:16:25.901153+0000","last_fullsized":"2024-04-15T15:16:25.901153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885679+0000","last_change":"2024-04-15T15:16:25.892453+0000","last_active":"2024-04-15T15:16:26.885679+0000","last_peered":"2024-04-15T15:16:26.885679+0000","last_clean":"2024-04-15T15:16:26.885679+0000","last_became_active":"2024-04-15T15:16:25.892168+0000","last_became_peered":"2024-04-15T15:16:25.892168+0000","last_unstale":"2024-04-15T15:16:26.885679+0000","last_undegraded":"2024-04-15T15:16:26.885679+0000","last_fullsized":"2024-04-15T15:16:26.885679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891390+0000","last_change":"2024-04-15T15:16:25.885032+0000","last_active":"2024-04-15T15:16:25.891390+0000","last_peered":"2024-04-15T15:16:25.891390+0000","last_clean":"2024-04-15T15:16:25.891390+0000","last_became_active":"2024-04-15T15:16:25.884397+0000","last_became_peered":"2024-04-15T15:16:25.884397+0000","last_unstale":"2024-04-15T15:16:25.891390+0000","last_undegraded":"2024-04-15T15:16:25.891390+0000","last_fullsized":"2024-04-15T15:16:25.891390+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:31.155865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885450+0000","last_change":"2024-04-15T15:16:25.882933+0000","last_active":"2024-04-15T15:16:26.885450+0000","last_peered":"2024-04-15T15:16:26.885450+0000","last_clean":"2024-04-15T15:16:26.885450+0000","last_became_active":"2024-04-15T15:16:25.882643+0000","last_became_peered":"2024-04-15T15:16:25.882643+0000","last_unstale":"2024-04-15T15:16:26.885450+0000","last_undegraded":"2024-04-15T15:16:26.885450+0000","last_fullsized":"2024-04-15T15:16:26.885450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891210+0000","last_change":"2024-04-15T15:16:25.890339+0000","last_active":"2024-04-15T15:16:25.891210+0000","last_peered":"2024-04-15T15:16:25.891210+0000","last_clean":"2024-04-15T15:16:25.891210+0000","last_became_active":"2024-04-15T15:16:25.889922+0000","last_became_peered":"2024-04-15T15:16:25.889922+0000","last_unstale":"2024-04-15T15:16:25.891210+0000","last_undegraded":"2024-04-15T15:16:25.891210+0000","last_fullsized":"2024-04-15T15:16:25.891210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885230+0000","last_change":"2024-04-15T15:16:25.890968+0000","last_active":"2024-04-15T15:16:26.885230+0000","last_peered":"2024-04-15T15:16:26.885230+0000","last_clean":"2024-04-15T15:16:26.885230+0000","last_became_active":"2024-04-15T15:16:25.890686+0000","last_became_peered":"2024-04-15T15:16:25.890686+0000","last_unstale":"2024-04-15T15:16:26.885230+0000","last_undegraded":"2024-04-15T15:16:26.885230+0000","last_fullsized":"2024-04-15T15:16:26.885230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904156+0000","last_change":"2024-04-15T15:16:25.900260+0000","last_active":"2024-04-15T15:16:25.904156+0000","last_peered":"2024-04-15T15:16:25.904156+0000","last_clean":"2024-04-15T15:16:25.904156+0000","last_became_active":"2024-04-15T15:16:25.899948+0000","last_became_peered":"2024-04-15T15:16:25.899948+0000","last_unstale":"2024-04-15T15:16:25.904156+0000","last_undegraded":"2024-04-15T15:16:25.904156+0000","last_fullsized":"2024-04-15T15:16:25.904156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884989+0000","last_change":"2024-04-15T15:16:25.884088+0000","last_active":"2024-04-15T15:16:26.884989+0000","last_peered":"2024-04-15T15:16:26.884989+0000","last_clean":"2024-04-15T15:16:26.884989+0000","last_became_active":"2024-04-15T15:16:25.883804+0000","last_became_peered":"2024-04-15T15:16:25.883804+0000","last_unstale":"2024-04-15T15:16:26.884989+0000","last_undegraded":"2024-04-15T15:16:26.884989+0000","last_fullsized":"2024-04-15T15:16:26.884989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885622+0000","last_change":"2024-04-15T15:16:25.879663+0000","last_active":"2024-04-15T15:16:26.885622+0000","last_peered":"2024-04-15T15:16:26.885622+0000","last_clean":"2024-04-15T15:16:26.885622+0000","last_became_active":"2024-04-15T15:16:25.879385+0000","last_became_peered":"2024-04-15T15:16:25.879385+0000","last_unstale":"2024-04-15T15:16:26.885622+0000","last_undegraded":"2024-04-15T15:16:26.885622+0000","last_fullsized":"2024-04-15T15:16:26.885622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888033+0000","last_change":"2024-04-15T15:16:25.886972+0000","last_active":"2024-04-15T15:16:25.888033+0000","last_peered":"2024-04-15T15:16:25.888033+0000","last_clean":"2024-04-15T15:16:25.888033+0000","last_became_active":"2024-04-15T15:16:25.886718+0000","last_became_peered":"2024-04-15T15:16:25.886718+0000","last_unstale":"2024-04-15T15:16:25.888033+0000","last_undegraded":"2024-04-15T15:16:25.888033+0000","last_fullsized":"2024-04-15T15:16:25.888033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885577+0000","last_change":"2024-04-15T15:16:24.886178+0000","last_active":"2024-04-15T15:16:26.885577+0000","last_peered":"2024-04-15T15:16:26.885577+0000","last_clean":"2024-04-15T15:16:26.885577+0000","last_became_active":"2024-04-15T15:16:24.885910+0000","last_became_peered":"2024-04-15T15:16:24.885910+0000","last_unstale":"2024-04-15T15:16:26.885577+0000","last_undegraded":"2024-04-15T15:16:26.885577+0000","last_fullsized":"2024-04-15T15:16:26.885577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903528+0000","last_change":"2024-04-15T15:16:25.902820+0000","last_active":"2024-04-15T15:16:25.903528+0000","last_peered":"2024-04-15T15:16:25.903528+0000","last_clean":"2024-04-15T15:16:25.903528+0000","last_became_active":"2024-04-15T15:16:25.902552+0000","last_became_peered":"2024-04-15T15:16:25.902552+0000","last_unstale":"2024-04-15T15:16:25.903528+0000","last_undegraded":"2024-04-15T15:16:25.903528+0000","last_fullsized":"2024-04-15T15:16:25.903528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898338+0000","last_change":"2024-04-15T15:16:24.886331+0000","last_active":"2024-04-15T15:16:25.898338+0000","last_peered":"2024-04-15T15:16:25.898338+0000","last_clean":"2024-04-15T15:16:25.898338+0000","last_became_active":"2024-04-15T15:16:24.886052+0000","last_became_peered":"2024-04-15T15:16:24.886052+0000","last_unstale":"2024-04-15T15:16:25.898338+0000","last_undegraded":"2024-04-15T15:16:25.898338+0000","last_fullsized":"2024-04-15T15:16:25.898338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.899883+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.899620+0000","last_became_peered":"2024-04-15T15:16:25.899620+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898694+0000","last_change":"2024-04-15T15:16:24.890646+0000","last_active":"2024-04-15T15:16:25.898694+0000","last_peered":"2024-04-15T15:16:25.898694+0000","last_clean":"2024-04-15T15:16:25.898694+0000","last_became_active":"2024-04-15T15:16:24.890362+0000","last_became_peered":"2024-04-15T15:16:24.890362+0000","last_unstale":"2024-04-15T15:16:25.898694+0000","last_undegraded":"2024-04-15T15:16:25.898694+0000","last_fullsized":"2024-04-15T15:16:25.898694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883847+0000","last_change":"2024-04-15T15:16:25.895423+0000","last_active":"2024-04-15T15:16:26.883847+0000","last_peered":"2024-04-15T15:16:26.883847+0000","last_clean":"2024-04-15T15:16:26.883847+0000","last_became_active":"2024-04-15T15:16:25.889905+0000","last_became_peered":"2024-04-15T15:16:25.889905+0000","last_unstale":"2024-04-15T15:16:26.883847+0000","last_undegraded":"2024-04-15T15:16:26.883847+0000","last_fullsized":"2024-04-15T15:16:26.883847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:50:15.175711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886111+0000","last_change":"2024-04-15T15:16:24.887352+0000","last_active":"2024-04-15T15:16:26.886111+0000","last_peered":"2024-04-15T15:16:26.886111+0000","last_clean":"2024-04-15T15:16:26.886111+0000","last_became_active":"2024-04-15T15:16:24.887054+0000","last_became_peered":"2024-04-15T15:16:24.887054+0000","last_unstale":"2024-04-15T15:16:26.886111+0000","last_undegraded":"2024-04-15T15:16:26.886111+0000","last_fullsized":"2024-04-15T15:16:26.886111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.882206+0000","last_change":"2024-04-15T15:16:25.881247+0000","last_active":"2024-04-15T15:16:25.882206+0000","last_peered":"2024-04-15T15:16:25.882206+0000","last_clean":"2024-04-15T15:16:25.882206+0000","last_became_active":"2024-04-15T15:16:25.880981+0000","last_became_peered":"2024-04-15T15:16:25.880981+0000","last_unstale":"2024-04-15T15:16:25.882206+0000","last_undegraded":"2024-04-15T15:16:25.882206+0000","last_fullsized":"2024-04-15T15:16:25.882206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903854+0000","last_change":"2024-04-15T15:16:24.891022+0000","last_active":"2024-04-15T15:16:25.903854+0000","last_peered":"2024-04-15T15:16:25.903854+0000","last_clean":"2024-04-15T15:16:25.903854+0000","last_became_active":"2024-04-15T15:16:24.890770+0000","last_became_peered":"2024-04-15T15:16:24.890770+0000","last_unstale":"2024-04-15T15:16:25.903854+0000","last_undegraded":"2024-04-15T15:16:25.903854+0000","last_fullsized":"2024-04-15T15:16:25.903854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901334+0000","last_change":"2024-04-15T15:16:25.893573+0000","last_active":"2024-04-15T15:16:25.901334+0000","last_peered":"2024-04-15T15:16:25.901334+0000","last_clean":"2024-04-15T15:16:25.901334+0000","last_became_active":"2024-04-15T15:16:25.893321+0000","last_became_peered":"2024-04-15T15:16:25.893321+0000","last_unstale":"2024-04-15T15:16:25.901334+0000","last_undegraded":"2024-04-15T15:16:25.901334+0000","last_fullsized":"2024-04-15T15:16:25.901334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885832+0000","last_change":"2024-04-15T15:16:25.904897+0000","last_active":"2024-04-15T15:16:26.885832+0000","last_peered":"2024-04-15T15:16:26.885832+0000","last_clean":"2024-04-15T15:16:26.885832+0000","last_became_active":"2024-04-15T15:16:25.904610+0000","last_became_peered":"2024-04-15T15:16:25.904610+0000","last_unstale":"2024-04-15T15:16:26.885832+0000","last_undegraded":"2024-04-15T15:16:26.885832+0000","last_fullsized":"2024-04-15T15:16:26.885832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876235+0000","last_change":"2024-04-15T15:16:24.891428+0000","last_active":"2024-04-15T15:16:25.876235+0000","last_peered":"2024-04-15T15:16:25.876235+0000","last_clean":"2024-04-15T15:16:25.876235+0000","last_became_active":"2024-04-15T15:16:24.891151+0000","last_became_peered":"2024-04-15T15:16:24.891151+0000","last_unstale":"2024-04-15T15:16:25.876235+0000","last_undegraded":"2024-04-15T15:16:25.876235+0000","last_fullsized":"2024-04-15T15:16:25.876235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:56:54.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883767+0000","last_change":"2024-04-15T15:16:25.898140+0000","last_active":"2024-04-15T15:16:26.883767+0000","last_peered":"2024-04-15T15:16:26.883767+0000","last_clean":"2024-04-15T15:16:26.883767+0000","last_became_active":"2024-04-15T15:16:25.897492+0000","last_became_peered":"2024-04-15T15:16:25.897492+0000","last_unstale":"2024-04-15T15:16:26.883767+0000","last_undegraded":"2024-04-15T15:16:26.883767+0000","last_fullsized":"2024-04-15T15:16:26.883767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884775+0000","last_change":"2024-04-15T15:16:24.884571+0000","last_active":"2024-04-15T15:16:26.884775+0000","last_peered":"2024-04-15T15:16:26.884775+0000","last_clean":"2024-04-15T15:16:26.884775+0000","last_became_active":"2024-04-15T15:16:24.884274+0000","last_became_peered":"2024-04-15T15:16:24.884274+0000","last_unstale":"2024-04-15T15:16:26.884775+0000","last_undegraded":"2024-04-15T15:16:26.884775+0000","last_fullsized":"2024-04-15T15:16:26.884775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.899098+0000","last_change":"2024-04-15T15:16:25.898346+0000","last_active":"2024-04-15T15:16:25.899098+0000","last_peered":"2024-04-15T15:16:25.899098+0000","last_clean":"2024-04-15T15:16:25.899098+0000","last_became_active":"2024-04-15T15:16:25.897145+0000","last_became_peered":"2024-04-15T15:16:25.897145+0000","last_unstale":"2024-04-15T15:16:25.899098+0000","last_undegraded":"2024-04-15T15:16:25.899098+0000","last_fullsized":"2024-04-15T15:16:25.899098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:54.775491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884135+0000","last_change":"2024-04-15T15:16:24.888936+0000","last_active":"2024-04-15T15:16:26.884135+0000","last_peered":"2024-04-15T15:16:26.884135+0000","last_clean":"2024-04-15T15:16:26.884135+0000","last_became_active":"2024-04-15T15:16:24.888050+0000","last_became_peered":"2024-04-15T15:16:24.888050+0000","last_unstale":"2024-04-15T15:16:26.884135+0000","last_undegraded":"2024-04-15T15:16:26.884135+0000","last_fullsized":"2024-04-15T15:16:26.884135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.886799+0000","last_change":"2024-04-15T15:16:24.887199+0000","last_active":"2024-04-15T15:16:25.886799+0000","last_peered":"2024-04-15T15:16:25.886799+0000","last_clean":"2024-04-15T15:16:25.886799+0000","last_became_active":"2024-04-15T15:16:24.886945+0000","last_became_peered":"2024-04-15T15:16:24.886945+0000","last_unstale":"2024-04-15T15:16:25.886799+0000","last_undegraded":"2024-04-15T15:16:25.886799+0000","last_fullsized":"2024-04-15T15:16:25.886799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.893867+0000","last_change":"2024-04-15T15:16:25.892203+0000","last_active":"2024-04-15T15:16:25.893867+0000","last_peered":"2024-04-15T15:16:25.893867+0000","last_clean":"2024-04-15T15:16:25.893867+0000","last_became_active":"2024-04-15T15:16:25.891899+0000","last_became_peered":"2024-04-15T15:16:25.891899+0000","last_unstale":"2024-04-15T15:16:25.893867+0000","last_undegraded":"2024-04-15T15:16:25.893867+0000","last_fullsized":"2024-04-15T15:16:25.893867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904566+0000","last_change":"2024-04-15T15:16:24.890265+0000","last_active":"2024-04-15T15:16:25.904566+0000","last_peered":"2024-04-15T15:16:25.904566+0000","last_clean":"2024-04-15T15:16:25.904566+0000","last_became_active":"2024-04-15T15:16:24.889963+0000","last_became_peered":"2024-04-15T15:16:24.889963+0000","last_unstale":"2024-04-15T15:16:25.904566+0000","last_undegraded":"2024-04-15T15:16:25.904566+0000","last_fullsized":"2024-04-15T15:16:25.904566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886503+0000","last_change":"2024-04-15T15:16:25.893921+0000","last_active":"2024-04-15T15:16:26.886503+0000","last_peered":"2024-04-15T15:16:26.886503+0000","last_clean":"2024-04-15T15:16:26.886503+0000","last_became_active":"2024-04-15T15:16:25.892154+0000","last_became_peered":"2024-04-15T15:16:25.892154+0000","last_unstale":"2024-04-15T15:16:26.886503+0000","last_undegraded":"2024-04-15T15:16:26.886503+0000","last_fullsized":"2024-04-15T15:16:26.886503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884185+0000","last_change":"2024-04-15T15:16:24.889313+0000","last_active":"2024-04-15T15:16:26.884185+0000","last_peered":"2024-04-15T15:16:26.884185+0000","last_clean":"2024-04-15T15:16:26.884185+0000","last_became_active":"2024-04-15T15:16:24.889037+0000","last_became_peered":"2024-04-15T15:16:24.889037+0000","last_unstale":"2024-04-15T15:16:26.884185+0000","last_undegraded":"2024-04-15T15:16:26.884185+0000","last_fullsized":"2024-04-15T15:16:26.884185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:58.773260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.889667+0000","last_change":"2024-04-15T15:16:25.888305+0000","last_active":"2024-04-15T15:16:25.889667+0000","last_peered":"2024-04-15T15:16:25.889667+0000","last_clean":"2024-04-15T15:16:25.889667+0000","last_became_active":"2024-04-15T15:16:25.888053+0000","last_became_peered":"2024-04-15T15:16:25.888053+0000","last_unstale":"2024-04-15T15:16:25.889667+0000","last_undegraded":"2024-04-15T15:16:25.889667+0000","last_fullsized":"2024-04-15T15:16:25.889667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884136+0000","last_change":"2024-04-15T15:16:24.882824+0000","last_active":"2024-04-15T15:16:26.884136+0000","last_peered":"2024-04-15T15:16:26.884136+0000","last_clean":"2024-04-15T15:16:26.884136+0000","last_became_active":"2024-04-15T15:16:24.882560+0000","last_became_peered":"2024-04-15T15:16:24.882560+0000","last_unstale":"2024-04-15T15:16:26.884136+0000","last_undegraded":"2024-04-15T15:16:26.884136+0000","last_fullsized":"2024-04-15T15:16:26.884136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:08.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883781+0000","last_change":"2024-04-15T15:16:25.883512+0000","last_active":"2024-04-15T15:16:26.883781+0000","last_peered":"2024-04-15T15:16:26.883781+0000","last_clean":"2024-04-15T15:16:26.883781+0000","last_became_active":"2024-04-15T15:16:25.883253+0000","last_became_peered":"2024-04-15T15:16:25.883253+0000","last_unstale":"2024-04-15T15:16:26.883781+0000","last_undegraded":"2024-04-15T15:16:26.883781+0000","last_fullsized":"2024-04-15T15:16:26.883781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.877572+0000","last_change":"2024-04-15T15:16:24.891194+0000","last_active":"2024-04-15T15:16:25.877572+0000","last_peered":"2024-04-15T15:16:25.877572+0000","last_clean":"2024-04-15T15:16:25.877572+0000","last_became_active":"2024-04-15T15:16:24.890944+0000","last_became_peered":"2024-04-15T15:16:24.890944+0000","last_unstale":"2024-04-15T15:16:25.877572+0000","last_undegraded":"2024-04-15T15:16:25.877572+0000","last_fullsized":"2024-04-15T15:16:25.877572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885147+0000","last_change":"2024-04-15T15:16:24.886029+0000","last_active":"2024-04-15T15:16:26.885147+0000","last_peered":"2024-04-15T15:16:26.885147+0000","last_clean":"2024-04-15T15:16:26.885147+0000","last_became_active":"2024-04-15T15:16:24.885730+0000","last_became_peered":"2024-04-15T15:16:24.885730+0000","last_unstale":"2024-04-15T15:16:26.885147+0000","last_undegraded":"2024-04-15T15:16:26.885147+0000","last_fullsized":"2024-04-15T15:16:26.885147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884020+0000","last_change":"2024-04-15T15:16:25.882916+0000","last_active":"2024-04-15T15:16:26.884020+0000","last_peered":"2024-04-15T15:16:26.884020+0000","last_clean":"2024-04-15T15:16:26.884020+0000","last_became_active":"2024-04-15T15:16:25.882630+0000","last_became_peered":"2024-04-15T15:16:25.882630+0000","last_unstale":"2024-04-15T15:16:26.884020+0000","last_undegraded":"2024-04-15T15:16:26.884020+0000","last_fullsized":"2024-04-15T15:16:26.884020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:24.310942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884117+0000","last_change":"2024-04-15T15:16:25.890929+0000","last_active":"2024-04-15T15:16:26.884117+0000","last_peered":"2024-04-15T15:16:26.884117+0000","last_clean":"2024-04-15T15:16:26.884117+0000","last_became_active":"2024-04-15T15:16:25.890256+0000","last_became_peered":"2024-04-15T15:16:25.890256+0000","last_unstale":"2024-04-15T15:16:26.884117+0000","last_undegraded":"2024-04-15T15:16:26.884117+0000","last_fullsized":"2024-04-15T15:16:26.884117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:11.834293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906758+0000","last_change":"2024-04-15T15:16:24.893507+0000","last_active":"2024-04-15T15:16:25.906758+0000","last_peered":"2024-04-15T15:16:25.906758+0000","last_clean":"2024-04-15T15:16:25.906758+0000","last_became_active":"2024-04-15T15:16:24.893262+0000","last_became_peered":"2024-04-15T15:16:24.893262+0000","last_unstale":"2024-04-15T15:16:25.906758+0000","last_undegraded":"2024-04-15T15:16:25.906758+0000","last_fullsized":"2024-04-15T15:16:25.906758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:15.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911705+0000","last_change":"2024-04-15T15:16:25.904384+0000","last_active":"2024-04-15T15:16:25.911705+0000","last_peered":"2024-04-15T15:16:25.911705+0000","last_clean":"2024-04-15T15:16:25.911705+0000","last_became_active":"2024-04-15T15:16:25.904095+0000","last_became_peered":"2024-04-15T15:16:25.904095+0000","last_unstale":"2024-04-15T15:16:25.911705+0000","last_undegraded":"2024-04-15T15:16:25.911705+0000","last_fullsized":"2024-04-15T15:16:25.911705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886583+0000","last_change":"2024-04-15T15:16:24.891423+0000","last_active":"2024-04-15T15:16:26.886583+0000","last_peered":"2024-04-15T15:16:26.886583+0000","last_clean":"2024-04-15T15:16:26.886583+0000","last_became_active":"2024-04-15T15:16:24.891048+0000","last_became_peered":"2024-04-15T15:16:24.891048+0000","last_unstale":"2024-04-15T15:16:26.886583+0000","last_undegraded":"2024-04-15T15:16:26.886583+0000","last_fullsized":"2024-04-15T15:16:26.886583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:47:10.367180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.894040+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.893781+0000","last_became_peered":"2024-04-15T15:16:25.893781+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.897493+0000","last_change":"2024-04-15T15:16:24.889598+0000","last_active":"2024-04-15T15:16:25.897493+0000","last_peered":"2024-04-15T15:16:25.897493+0000","last_clean":"2024-04-15T15:16:25.897493+0000","last_became_active":"2024-04-15T15:16:24.889298+0000","last_became_peered":"2024-04-15T15:16:24.889298+0000","last_unstale":"2024-04-15T15:16:25.897493+0000","last_undegraded":"2024-04-15T15:16:25.897493+0000","last_fullsized":"2024-04-15T15:16:25.897493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:54:18.385006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891485+0000","last_change":"2024-04-15T15:16:25.884522+0000","last_active":"2024-04-15T15:16:25.891485+0000","last_peered":"2024-04-15T15:16:25.891485+0000","last_clean":"2024-04-15T15:16:25.891485+0000","last_became_active":"2024-04-15T15:16:25.883126+0000","last_became_peered":"2024-04-15T15:16:25.883126+0000","last_unstale":"2024-04-15T15:16:25.891485+0000","last_undegraded":"2024-04-15T15:16:25.891485+0000","last_fullsized":"2024-04-15T15:16:25.891485+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:03.969456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876163+0000","last_change":"2024-04-15T15:16:24.890031+0000","last_active":"2024-04-15T15:16:25.876163+0000","last_peered":"2024-04-15T15:16:25.876163+0000","last_clean":"2024-04-15T15:16:25.876163+0000","last_became_active":"2024-04-15T15:16:24.889203+0000","last_became_peered":"2024-04-15T15:16:24.889203+0000","last_unstale":"2024-04-15T15:16:25.876163+0000","last_undegraded":"2024-04-15T15:16:25.876163+0000","last_fullsized":"2024-04-15T15:16:25.876163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:18.919033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891401+0000","last_change":"2024-04-15T15:16:24.877420+0000","last_active":"2024-04-15T15:16:25.891401+0000","last_peered":"2024-04-15T15:16:25.891401+0000","last_clean":"2024-04-15T15:16:25.891401+0000","last_became_active":"2024-04-15T15:16:24.877149+0000","last_became_peered":"2024-04-15T15:16:24.877149+0000","last_unstale":"2024-04-15T15:16:25.891401+0000","last_undegraded":"2024-04-15T15:16:25.891401+0000","last_fullsized":"2024-04-15T15:16:25.891401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:48.237807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.890064+0000","last_change":"2024-04-15T15:16:25.887625+0000","last_active":"2024-04-15T15:16:25.890064+0000","last_peered":"2024-04-15T15:16:25.890064+0000","last_clean":"2024-04-15T15:16:25.890064+0000","last_became_active":"2024-04-15T15:16:25.887362+0000","last_became_peered":"2024-04-15T15:16:25.887362+0000","last_unstale":"2024-04-15T15:16:25.890064+0000","last_undegraded":"2024-04-15T15:16:25.890064+0000","last_fullsized":"2024-04-15T15:16:25.890064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872610+0000","last_change":"2024-04-15T15:16:24.884150+0000","last_active":"2024-04-15T15:16:25.872610+0000","last_peered":"2024-04-15T15:16:25.872610+0000","last_clean":"2024-04-15T15:16:25.872610+0000","last_became_active":"2024-04-15T15:16:24.883908+0000","last_became_peered":"2024-04-15T15:16:24.883908+0000","last_unstale":"2024-04-15T15:16:25.872610+0000","last_undegraded":"2024-04-15T15:16:25.872610+0000","last_fullsized":"2024-04-15T15:16:25.872610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:58:35.147306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903908+0000","last_change":"2024-04-15T15:16:25.897701+0000","last_active":"2024-04-15T15:16:25.903908+0000","last_peered":"2024-04-15T15:16:25.903908+0000","last_clean":"2024-04-15T15:16:25.903908+0000","last_became_active":"2024-04-15T15:16:25.896891+0000","last_became_peered":"2024-04-15T15:16:25.896891+0000","last_unstale":"2024-04-15T15:16:25.903908+0000","last_undegraded":"2024-04-15T15:16:25.903908+0000","last_fullsized":"2024-04-15T15:16:25.903908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884030+0000","last_change":"2024-04-15T15:16:25.899465+0000","last_active":"2024-04-15T15:16:26.884030+0000","last_peered":"2024-04-15T15:16:26.884030+0000","last_clean":"2024-04-15T15:16:26.884030+0000","last_became_active":"2024-04-15T15:16:25.899205+0000","last_became_peered":"2024-04-15T15:16:25.899205+0000","last_unstale":"2024-04-15T15:16:26.884030+0000","last_undegraded":"2024-04-15T15:16:26.884030+0000","last_fullsized":"2024-04-15T15:16:26.884030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:42:16.777997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872575+0000","last_change":"2024-04-15T15:16:24.882714+0000","last_active":"2024-04-15T15:16:25.872575+0000","last_peered":"2024-04-15T15:16:25.872575+0000","last_clean":"2024-04-15T15:16:25.872575+0000","last_became_active":"2024-04-15T15:16:24.882460+0000","last_became_peered":"2024-04-15T15:16:24.882460+0000","last_unstale":"2024-04-15T15:16:25.872575+0000","last_undegraded":"2024-04-15T15:16:25.872575+0000","last_fullsized":"2024-04-15T15:16:25.872575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:11.251738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898832+0000","last_change":"2024-04-15T15:16:25.897148+0000","last_active":"2024-04-15T15:16:25.898832+0000","last_peered":"2024-04-15T15:16:25.898832+0000","last_clean":"2024-04-15T15:16:25.898832+0000","last_became_active":"2024-04-15T15:16:25.896866+0000","last_became_peered":"2024-04-15T15:16:25.896866+0000","last_unstale":"2024-04-15T15:16:25.898832+0000","last_undegraded":"2024-04-15T15:16:25.898832+0000","last_fullsized":"2024-04-15T15:16:25.898832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902029+0000","last_change":"2024-04-15T15:16:24.892956+0000","last_active":"2024-04-15T15:16:25.902029+0000","last_peered":"2024-04-15T15:16:25.902029+0000","last_clean":"2024-04-15T15:16:25.902029+0000","last_became_active":"2024-04-15T15:16:24.892664+0000","last_became_peered":"2024-04-15T15:16:24.892664+0000","last_unstale":"2024-04-15T15:16:25.902029+0000","last_undegraded":"2024-04-15T15:16:25.902029+0000","last_fullsized":"2024-04-15T15:16:25.902029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:56.013415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.873000+0000","last_change":"2024-04-15T15:16:24.890594+0000","last_active":"2024-04-15T15:16:25.873000+0000","last_peered":"2024-04-15T15:16:25.873000+0000","last_clean":"2024-04-15T15:16:25.873000+0000","last_became_active":"2024-04-15T15:16:24.890305+0000","last_became_peered":"2024-04-15T15:16:24.890305+0000","last_unstale":"2024-04-15T15:16:25.873000+0000","last_undegraded":"2024-04-15T15:16:25.873000+0000","last_fullsized":"2024-04-15T15:16:25.873000+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:34.555529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884060+0000","last_change":"2024-04-15T15:16:25.893319+0000","last_active":"2024-04-15T15:16:26.884060+0000","last_peered":"2024-04-15T15:16:26.884060+0000","last_clean":"2024-04-15T15:16:26.884060+0000","last_became_active":"2024-04-15T15:16:25.892794+0000","last_became_peered":"2024-04-15T15:16:25.892794+0000","last_unstale":"2024-04-15T15:16:26.884060+0000","last_undegraded":"2024-04-15T15:16:26.884060+0000","last_fullsized":"2024-04-15T15:16:26.884060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:42.664115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876776+0000","last_change":"2024-04-15T15:16:24.884802+0000","last_active":"2024-04-15T15:16:25.876776+0000","last_peered":"2024-04-15T15:16:25.876776+0000","last_clean":"2024-04-15T15:16:25.876776+0000","last_became_active":"2024-04-15T15:16:24.884519+0000","last_became_peered":"2024-04-15T15:16:24.884519+0000","last_unstale":"2024-04-15T15:16:25.876776+0000","last_undegraded":"2024-04-15T15:16:25.876776+0000","last_fullsized":"2024-04-15T15:16:25.876776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888791+0000","last_change":"2024-04-15T15:16:25.883444+0000","last_active":"2024-04-15T15:16:25.888791+0000","last_peered":"2024-04-15T15:16:25.888791+0000","last_clean":"2024-04-15T15:16:25.888791+0000","last_became_active":"2024-04-15T15:16:25.883191+0000","last_became_peered":"2024-04-15T15:16:25.883191+0000","last_unstale":"2024-04-15T15:16:25.888791+0000","last_undegraded":"2024-04-15T15:16:25.888791+0000","last_fullsized":"2024-04-15T15:16:25.888791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.875590+0000","last_change":"2024-04-15T15:16:24.886148+0000","last_active":"2024-04-15T15:16:25.875590+0000","last_peered":"2024-04-15T15:16:25.875590+0000","last_clean":"2024-04-15T15:16:25.875590+0000","last_became_active":"2024-04-15T15:16:24.885865+0000","last_became_peered":"2024-04-15T15:16:24.885865+0000","last_unstale":"2024-04-15T15:16:25.875590+0000","last_undegraded":"2024-04-15T15:16:25.875590+0000","last_fullsized":"2024-04-15T15:16:25.875590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888002+0000","last_change":"2024-04-15T15:16:25.886178+0000","last_active":"2024-04-15T15:16:25.888002+0000","last_peered":"2024-04-15T15:16:25.888002+0000","last_clean":"2024-04-15T15:16:25.888002+0000","last_became_active":"2024-04-15T15:16:25.885915+0000","last_became_peered":"2024-04-15T15:16:25.885915+0000","last_unstale":"2024-04-15T15:16:25.888002+0000","last_undegraded":"2024-04-15T15:16:25.888002+0000","last_fullsized":"2024-04-15T15:16:25.888002+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:33.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885719+0000","last_change":"2024-04-15T15:16:24.883958+0000","last_active":"2024-04-15T15:16:26.885719+0000","last_peered":"2024-04-15T15:16:26.885719+0000","last_clean":"2024-04-15T15:16:26.885719+0000","last_became_active":"2024-04-15T15:16:24.883528+0000","last_became_peered":"2024-04-15T15:16:24.883528+0000","last_unstale":"2024-04-15T15:16:26.885719+0000","last_undegraded":"2024-04-15T15:16:26.885719+0000","last_fullsized":"2024-04-15T15:16:26.885719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885492+0000","last_change":"2024-04-15T15:16:25.889861+0000","last_active":"2024-04-15T15:16:26.885492+0000","last_peered":"2024-04-15T15:16:26.885492+0000","last_clean":"2024-04-15T15:16:26.885492+0000","last_became_active":"2024-04-15T15:16:25.889591+0000","last_became_peered":"2024-04-15T15:16:25.889591+0000","last_unstale":"2024-04-15T15:16:26.885492+0000","last_undegraded":"2024-04-15T15:16:26.885492+0000","last_fullsized":"2024-04-15T15:16:26.885492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884867+0000","last_change":"2024-04-15T15:16:24.891429+0000","last_active":"2024-04-15T15:16:26.884867+0000","last_peered":"2024-04-15T15:16:26.884867+0000","last_clean":"2024-04-15T15:16:26.884867+0000","last_became_active":"2024-04-15T15:16:24.891111+0000","last_became_peered":"2024-04-15T15:16:24.891111+0000","last_unstale":"2024-04-15T15:16:26.884867+0000","last_undegraded":"2024-04-15T15:16:26.884867+0000","last_fullsized":"2024-04-15T15:16:26.884867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883631+0000","last_change":"2024-04-15T15:16:24.887452+0000","last_active":"2024-04-15T15:16:26.883631+0000","last_peered":"2024-04-15T15:16:26.883631+0000","last_clean":"2024-04-15T15:16:26.883631+0000","last_became_active":"2024-04-15T15:16:24.887136+0000","last_became_peered":"2024-04-15T15:16:24.887136+0000","last_unstale":"2024-04-15T15:16:26.883631+0000","last_undegraded":"2024-04-15T15:16:26.883631+0000","last_fullsized":"2024-04-15T15:16:26.883631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:24.817169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906655+0000","last_change":"2024-04-15T15:16:25.901054+0000","last_active":"2024-04-15T15:16:25.906655+0000","last_peered":"2024-04-15T15:16:25.906655+0000","last_clean":"2024-04-15T15:16:25.906655+0000","last_became_active":"2024-04-15T15:16:25.900784+0000","last_became_peered":"2024-04-15T15:16:25.900784+0000","last_unstale":"2024-04-15T15:16:25.906655+0000","last_undegraded":"2024-04-15T15:16:25.906655+0000","last_fullsized":"2024-04-15T15:16:25.906655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-15T15:16:20.700430+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:16:20.700430+0000","last_peered":"2024-04-15T15:16:20.700430+0000","last_clean":"2024-04-15T15:16:20.700430+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:16:20.700430+0000","last_undegraded":"2024-04-15T15:16:20.700430+0000","last_fullsized":"2024-04-15T15:16:20.700430+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":108,"acting":108,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":56,"acting":56,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":758052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":189373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":778129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:29.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.370+0000 7fa5dc035640 1 Processor -- start 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.370+0000 7fa5dc035640 1 -- start start 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5dc035640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa5d415e830 0x7fa5d40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5dc035640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5d40b53c0 0x7fa5d40b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5dc035640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d4067e30 con 0x7fa5d415ec10 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5dc035640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d40b3c10 con 0x7fa5d40b53c0 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5dc035640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d40b3ef0 con 0x7fa5d415e830 2024-04-15T15:16:29.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5d40b53c0 0x7fa5d40b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa5d40b53c0 0x7fa5d40b5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49790/0 (socket says 172.21.15.70:49790) 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 -- 172.21.15.70:0/4011766421 learned_addr learned my addr 172.21.15.70:0/4011766421 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5d9dd1640 1 --2- 172.21.15.70:0/4011766421 >> v2:172.21.15.96:3300/0 conn(0x7fa5d415e830 0x7fa5d40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 -- 172.21.15.70:0/4011766421 >> v2:172.21.15.96:3300/0 conn(0x7fa5d415e830 msgr2=0x7fa5d40b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 --2- 172.21.15.70:0/4011766421 >> v2:172.21.15.96:3300/0 conn(0x7fa5d415e830 0x7fa5d40b4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 -- 172.21.15.70:0/4011766421 >> v1:172.21.15.96:6789/0 conn(0x7fa5d415ec10 legacy=0x7fa5d4067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.371+0000 7fa5da5d2640 1 -- 172.21.15.70:0/4011766421 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d40b41d0 con 0x7fa5d40b53c0 2024-04-15T15:16:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.372+0000 7fa5da5d2640 1 --2- 172.21.15.70:0/4011766421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d40b53c0 0x7fa5d40b5870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4073d70 tx=0x7fa5cc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcfbb32ead9a329 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.372+0000 7fa5d8dcf640 1 -- 172.21.15.70:0/4011766421 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc02e070 con 0x7fa5d40b53c0 2024-04-15T15:16:29.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.372+0000 7fa5d8dcf640 1 -- 172.21.15.70:0/4011766421 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5cc037070 con 0x7fa5d40b53c0 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.372+0000 7fa5d8dcf640 1 -- 172.21.15.70:0/4011766421 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc031a00 con 0x7fa5d40b53c0 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 -- 172.21.15.70:0/4011766421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d40b53c0 msgr2=0x7fa5d40b5870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 --2- 172.21.15.70:0/4011766421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d40b53c0 0x7fa5d40b5870 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4073d70 tx=0x7fa5cc02b880 comp rx=0 tx=0).stop 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 -- 172.21.15.70:0/4011766421 shutdown_connections 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 --2- 172.21.15.70:0/4011766421 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d40b53c0 0x7fa5d40b5870 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 --2- 172.21.15.70:0/4011766421 >> v2:172.21.15.96:3300/0 conn(0x7fa5d415e830 0x7fa5d40b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 -- 172.21.15.70:0/4011766421 >> 172.21.15.70:0/4011766421 conn(0x7fa5d4073360 msgr2=0x7fa5d406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 -- 172.21.15.70:0/4011766421 shutdown_connections 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.373+0000 7fa5dc035640 1 -- 172.21.15.70:0/4011766421 wait complete. 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.374+0000 7fa5dc035640 1 Processor -- start 2024-04-15T15:16:29.376 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.374+0000 7fa5dc035640 1 -- start start 2024-04-15T15:16:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.374+0000 7fa5dc035640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5d40b53c0 0x7fa5d41f72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5d4158bd0 0x7fa5d41e5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d415ff60 con 0x7fa5d415e830 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d40b4790 con 0x7fa5d40b53c0 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d41600e0 con 0x7fa5d4158bd0 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49800/0 (socket says 172.21.15.70:49800) 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 -- 172.21.15.70:0/2193085818 learned_addr learned my addr 172.21.15.70:0/2193085818 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 -- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5d4158bd0 msgr2=0x7fa5d41e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d9dd1640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5d40b53c0 0x7fa5d41f72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5d4158bd0 0x7fa5d41e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 -- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5d40b53c0 msgr2=0x7fa5d41f72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5d40b53c0 0x7fa5d41f72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 -- 172.21.15.70:0/2193085818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d4156270 con 0x7fa5d415e830 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5d95d0640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400b3e0 tx=0x7fa5c400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4015020 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c4011830 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d4156550 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4011c20 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.375+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d4156b10 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.376+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d40689e0 con 0x7fa5d415e830 2024-04-15T15:16:29.380 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.377+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fa5c4018020 con 0x7fa5d415e830 2024-04-15T15:16:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.377+0000 7fa5ca7fc640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 0x7fa5b806ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.377+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fa5c408bc50 con 0x7fa5d415e830 2024-04-15T15:16:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.377+0000 7fa5d9dd1640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 0x7fa5b806ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.381 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.378+0000 7fa5d9dd1640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 0x7fa5b806ede0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0008ef0 tx=0x7fa5d0008440 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.381+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa5c4091020 con 0x7fa5d415e830 2024-04-15T15:16:29.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.553+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa5d406ff10 con 0x7fa5b806c920 2024-04-15T15:16:29.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.558+0000 7fa5ca7fc640 1 -- 172.21.15.70:0/2193085818 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+249848 (secure 0 0 0) 0x7fa5d406ff10 con 0x7fa5b806c920 2024-04-15T15:16:29.563 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:29.564 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 msgr2=0x7fa5b806ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 0x7fa5b806ede0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0008ef0 tx=0x7fa5d0008440 comp rx=0 tx=0).stop 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 msgr2=0x7fa5d41559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400b3e0 tx=0x7fa5c400b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 shutdown_connections 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa5d4158bd0 0x7fa5d41e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa5b806c920 0x7fa5b806ede0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa5d415e830 0x7fa5d41559b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 --2- 172.21.15.70:0/2193085818 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa5d40b53c0 0x7fa5d41f72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 >> 172.21.15.70:0/2193085818 conn(0x7fa5d4073360 msgr2=0x7fa5d40afec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 shutdown_connections 2024-04-15T15:16:29.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.561+0000 7fa5dc035640 1 -- 172.21.15.70:0/2193085818 wait complete. 2024-04-15T15:16:29.574 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-04-15T15:16:28.085226+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":191,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":166,"acting":166,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":127,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237408,"kb_used_data":4596,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737312,"statfs":{"total":773094113280,"available":772851007488,"internally_reserved":0,"allocated":4706304,"data_stored":2665647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.219711"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884133+0000","last_change":"2024-04-15T15:16:25.883054+0000","last_active":"2024-04-15T15:16:26.884133+0000","last_peered":"2024-04-15T15:16:26.884133+0000","last_clean":"2024-04-15T15:16:26.884133+0000","last_became_active":"2024-04-15T15:16:25.882744+0000","last_became_peered":"2024-04-15T15:16:25.882744+0000","last_unstale":"2024-04-15T15:16:26.884133+0000","last_undegraded":"2024-04-15T15:16:26.884133+0000","last_fullsized":"2024-04-15T15:16:26.884133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:06:55.204459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883810+0000","last_change":"2024-04-15T15:16:25.888729+0000","last_active":"2024-04-15T15:16:26.883810+0000","last_peered":"2024-04-15T15:16:26.883810+0000","last_clean":"2024-04-15T15:16:26.883810+0000","last_became_active":"2024-04-15T15:16:25.888450+0000","last_became_peered":"2024-04-15T15:16:25.888450+0000","last_unstale":"2024-04-15T15:16:26.883810+0000","last_undegraded":"2024-04-15T15:16:26.883810+0000","last_fullsized":"2024-04-15T15:16:26.883810+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902977+0000","last_change":"2024-04-15T15:16:25.902269+0000","last_active":"2024-04-15T15:16:25.902977+0000","last_peered":"2024-04-15T15:16:25.902977+0000","last_clean":"2024-04-15T15:16:25.902977+0000","last_became_active":"2024-04-15T15:16:25.901972+0000","last_became_peered":"2024-04-15T15:16:25.901972+0000","last_unstale":"2024-04-15T15:16:25.902977+0000","last_undegraded":"2024-04-15T15:16:25.902977+0000","last_fullsized":"2024-04-15T15:16:25.902977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898887+0000","last_change":"2024-04-15T15:16:25.897033+0000","last_active":"2024-04-15T15:16:25.898887+0000","last_peered":"2024-04-15T15:16:25.898887+0000","last_clean":"2024-04-15T15:16:25.898887+0000","last_became_active":"2024-04-15T15:16:25.896432+0000","last_became_peered":"2024-04-15T15:16:25.896432+0000","last_unstale":"2024-04-15T15:16:25.898887+0000","last_undegraded":"2024-04-15T15:16:25.898887+0000","last_fullsized":"2024-04-15T15:16:25.898887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:19.565523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891565+0000","last_change":"2024-04-15T15:16:25.889069+0000","last_active":"2024-04-15T15:16:25.891565+0000","last_peered":"2024-04-15T15:16:25.891565+0000","last_clean":"2024-04-15T15:16:25.891565+0000","last_became_active":"2024-04-15T15:16:25.888793+0000","last_became_peered":"2024-04-15T15:16:25.888793+0000","last_unstale":"2024-04-15T15:16:25.891565+0000","last_undegraded":"2024-04-15T15:16:25.891565+0000","last_fullsized":"2024-04-15T15:16:25.891565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:48.611396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888862+0000","last_change":"2024-04-15T15:16:25.887072+0000","last_active":"2024-04-15T15:16:25.888862+0000","last_peered":"2024-04-15T15:16:25.888862+0000","last_clean":"2024-04-15T15:16:25.888862+0000","last_became_active":"2024-04-15T15:16:25.886821+0000","last_became_peered":"2024-04-15T15:16:25.886821+0000","last_unstale":"2024-04-15T15:16:25.888862+0000","last_undegraded":"2024-04-15T15:16:25.888862+0000","last_fullsized":"2024-04-15T15:16:25.888862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904107+0000","last_change":"2024-04-15T15:16:25.900378+0000","last_active":"2024-04-15T15:16:25.904107+0000","last_peered":"2024-04-15T15:16:25.904107+0000","last_clean":"2024-04-15T15:16:25.904107+0000","last_became_active":"2024-04-15T15:16:25.900071+0000","last_became_peered":"2024-04-15T15:16:25.900071+0000","last_unstale":"2024-04-15T15:16:25.904107+0000","last_undegraded":"2024-04-15T15:16:25.904107+0000","last_fullsized":"2024-04-15T15:16:25.904107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:43.288974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883618+0000","last_change":"2024-04-15T15:16:25.893507+0000","last_active":"2024-04-15T15:16:26.883618+0000","last_peered":"2024-04-15T15:16:26.883618+0000","last_clean":"2024-04-15T15:16:26.883618+0000","last_became_active":"2024-04-15T15:16:25.893005+0000","last_became_peered":"2024-04-15T15:16:25.893005+0000","last_unstale":"2024-04-15T15:16:26.883618+0000","last_undegraded":"2024-04-15T15:16:26.883618+0000","last_fullsized":"2024-04-15T15:16:26.883618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.895340+0000","last_change":"2024-04-15T15:16:25.894870+0000","last_active":"2024-04-15T15:16:25.895340+0000","last_peered":"2024-04-15T15:16:25.895340+0000","last_clean":"2024-04-15T15:16:25.895340+0000","last_became_active":"2024-04-15T15:16:25.894343+0000","last_became_peered":"2024-04-15T15:16:25.894343+0000","last_unstale":"2024-04-15T15:16:25.895340+0000","last_undegraded":"2024-04-15T15:16:25.895340+0000","last_fullsized":"2024-04-15T15:16:25.895340+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904248+0000","last_change":"2024-04-15T15:16:25.900279+0000","last_active":"2024-04-15T15:16:25.904248+0000","last_peered":"2024-04-15T15:16:25.904248+0000","last_clean":"2024-04-15T15:16:25.904248+0000","last_became_active":"2024-04-15T15:16:25.899969+0000","last_became_peered":"2024-04-15T15:16:25.899969+0000","last_unstale":"2024-04-15T15:16:25.904248+0000","last_undegraded":"2024-04-15T15:16:25.904248+0000","last_fullsized":"2024-04-15T15:16:25.904248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:19:11.196336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911723+0000","last_change":"2024-04-15T15:16:25.904761+0000","last_active":"2024-04-15T15:16:25.911723+0000","last_peered":"2024-04-15T15:16:25.911723+0000","last_clean":"2024-04-15T15:16:25.911723+0000","last_became_active":"2024-04-15T15:16:25.904477+0000","last_became_peered":"2024-04-15T15:16:25.904477+0000","last_unstale":"2024-04-15T15:16:25.911723+0000","last_undegraded":"2024-04-15T15:16:25.911723+0000","last_fullsized":"2024-04-15T15:16:25.911723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884502+0000","last_change":"2024-04-15T15:16:25.882646+0000","last_active":"2024-04-15T15:16:25.884502+0000","last_peered":"2024-04-15T15:16:25.884502+0000","last_clean":"2024-04-15T15:16:25.884502+0000","last_became_active":"2024-04-15T15:16:25.882375+0000","last_became_peered":"2024-04-15T15:16:25.882375+0000","last_unstale":"2024-04-15T15:16:25.884502+0000","last_undegraded":"2024-04-15T15:16:25.884502+0000","last_fullsized":"2024-04-15T15:16:25.884502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906634+0000","last_change":"2024-04-15T15:16:25.901003+0000","last_active":"2024-04-15T15:16:25.906634+0000","last_peered":"2024-04-15T15:16:25.906634+0000","last_clean":"2024-04-15T15:16:25.906634+0000","last_became_active":"2024-04-15T15:16:25.900729+0000","last_became_peered":"2024-04-15T15:16:25.900729+0000","last_unstale":"2024-04-15T15:16:25.906634+0000","last_undegraded":"2024-04-15T15:16:25.906634+0000","last_fullsized":"2024-04-15T15:16:25.906634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:34.769819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885365+0000","last_change":"2024-04-15T15:16:25.899260+0000","last_active":"2024-04-15T15:16:26.885365+0000","last_peered":"2024-04-15T15:16:26.885365+0000","last_clean":"2024-04-15T15:16:26.885365+0000","last_became_active":"2024-04-15T15:16:25.898979+0000","last_became_peered":"2024-04-15T15:16:25.898979+0000","last_unstale":"2024-04-15T15:16:26.885365+0000","last_undegraded":"2024-04-15T15:16:26.885365+0000","last_fullsized":"2024-04-15T15:16:26.885365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904064+0000","last_change":"2024-04-15T15:16:25.899744+0000","last_active":"2024-04-15T15:16:25.904064+0000","last_peered":"2024-04-15T15:16:25.904064+0000","last_clean":"2024-04-15T15:16:25.904064+0000","last_became_active":"2024-04-15T15:16:25.899472+0000","last_became_peered":"2024-04-15T15:16:25.899472+0000","last_unstale":"2024-04-15T15:16:25.904064+0000","last_undegraded":"2024-04-15T15:16:25.904064+0000","last_fullsized":"2024-04-15T15:16:25.904064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885281+0000","last_change":"2024-04-15T15:16:25.892123+0000","last_active":"2024-04-15T15:16:26.885281+0000","last_peered":"2024-04-15T15:16:26.885281+0000","last_clean":"2024-04-15T15:16:26.885281+0000","last_became_active":"2024-04-15T15:16:25.891853+0000","last_became_peered":"2024-04-15T15:16:25.891853+0000","last_unstale":"2024-04-15T15:16:26.885281+0000","last_undegraded":"2024-04-15T15:16:26.885281+0000","last_fullsized":"2024-04-15T15:16:26.885281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884688+0000","last_change":"2024-04-15T15:16:25.880039+0000","last_active":"2024-04-15T15:16:25.884688+0000","last_peered":"2024-04-15T15:16:25.884688+0000","last_clean":"2024-04-15T15:16:25.884688+0000","last_became_active":"2024-04-15T15:16:25.879753+0000","last_became_peered":"2024-04-15T15:16:25.879753+0000","last_unstale":"2024-04-15T15:16:25.884688+0000","last_undegraded":"2024-04-15T15:16:25.884688+0000","last_fullsized":"2024-04-15T15:16:25.884688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:58:15.988048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885269+0000","last_change":"2024-04-15T15:16:25.897291+0000","last_active":"2024-04-15T15:16:26.885269+0000","last_peered":"2024-04-15T15:16:26.885269+0000","last_clean":"2024-04-15T15:16:26.885269+0000","last_became_active":"2024-04-15T15:16:25.896977+0000","last_became_peered":"2024-04-15T15:16:25.896977+0000","last_unstale":"2024-04-15T15:16:26.885269+0000","last_undegraded":"2024-04-15T15:16:26.885269+0000","last_fullsized":"2024-04-15T15:16:26.885269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901153+0000","last_change":"2024-04-15T15:16:25.889177+0000","last_active":"2024-04-15T15:16:25.901153+0000","last_peered":"2024-04-15T15:16:25.901153+0000","last_clean":"2024-04-15T15:16:25.901153+0000","last_became_active":"2024-04-15T15:16:25.888843+0000","last_became_peered":"2024-04-15T15:16:25.888843+0000","last_unstale":"2024-04-15T15:16:25.901153+0000","last_undegraded":"2024-04-15T15:16:25.901153+0000","last_fullsized":"2024-04-15T15:16:25.901153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885679+0000","last_change":"2024-04-15T15:16:25.892453+0000","last_active":"2024-04-15T15:16:26.885679+0000","last_peered":"2024-04-15T15:16:26.885679+0000","last_clean":"2024-04-15T15:16:26.885679+0000","last_became_active":"2024-04-15T15:16:25.892168+0000","last_became_peered":"2024-04-15T15:16:25.892168+0000","last_unstale":"2024-04-15T15:16:26.885679+0000","last_undegraded":"2024-04-15T15:16:26.885679+0000","last_fullsized":"2024-04-15T15:16:26.885679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891390+0000","last_change":"2024-04-15T15:16:25.885032+0000","last_active":"2024-04-15T15:16:25.891390+0000","last_peered":"2024-04-15T15:16:25.891390+0000","last_clean":"2024-04-15T15:16:25.891390+0000","last_became_active":"2024-04-15T15:16:25.884397+0000","last_became_peered":"2024-04-15T15:16:25.884397+0000","last_unstale":"2024-04-15T15:16:25.891390+0000","last_undegraded":"2024-04-15T15:16:25.891390+0000","last_fullsized":"2024-04-15T15:16:25.891390+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:31.155865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885450+0000","last_change":"2024-04-15T15:16:25.882933+0000","last_active":"2024-04-15T15:16:26.885450+0000","last_peered":"2024-04-15T15:16:26.885450+0000","last_clean":"2024-04-15T15:16:26.885450+0000","last_became_active":"2024-04-15T15:16:25.882643+0000","last_became_peered":"2024-04-15T15:16:25.882643+0000","last_unstale":"2024-04-15T15:16:26.885450+0000","last_undegraded":"2024-04-15T15:16:26.885450+0000","last_fullsized":"2024-04-15T15:16:26.885450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891210+0000","last_change":"2024-04-15T15:16:25.890339+0000","last_active":"2024-04-15T15:16:25.891210+0000","last_peered":"2024-04-15T15:16:25.891210+0000","last_clean":"2024-04-15T15:16:25.891210+0000","last_became_active":"2024-04-15T15:16:25.889922+0000","last_became_peered":"2024-04-15T15:16:25.889922+0000","last_unstale":"2024-04-15T15:16:25.891210+0000","last_undegraded":"2024-04-15T15:16:25.891210+0000","last_fullsized":"2024-04-15T15:16:25.891210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885230+0000","last_change":"2024-04-15T15:16:25.890968+0000","last_active":"2024-04-15T15:16:26.885230+0000","last_peered":"2024-04-15T15:16:26.885230+0000","last_clean":"2024-04-15T15:16:26.885230+0000","last_became_active":"2024-04-15T15:16:25.890686+0000","last_became_peered":"2024-04-15T15:16:25.890686+0000","last_unstale":"2024-04-15T15:16:26.885230+0000","last_undegraded":"2024-04-15T15:16:26.885230+0000","last_fullsized":"2024-04-15T15:16:26.885230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904156+0000","last_change":"2024-04-15T15:16:25.900260+0000","last_active":"2024-04-15T15:16:25.904156+0000","last_peered":"2024-04-15T15:16:25.904156+0000","last_clean":"2024-04-15T15:16:25.904156+0000","last_became_active":"2024-04-15T15:16:25.899948+0000","last_became_peered":"2024-04-15T15:16:25.899948+0000","last_unstale":"2024-04-15T15:16:25.904156+0000","last_undegraded":"2024-04-15T15:16:25.904156+0000","last_fullsized":"2024-04-15T15:16:25.904156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884989+0000","last_change":"2024-04-15T15:16:25.884088+0000","last_active":"2024-04-15T15:16:26.884989+0000","last_peered":"2024-04-15T15:16:26.884989+0000","last_clean":"2024-04-15T15:16:26.884989+0000","last_became_active":"2024-04-15T15:16:25.883804+0000","last_became_peered":"2024-04-15T15:16:25.883804+0000","last_unstale":"2024-04-15T15:16:26.884989+0000","last_undegraded":"2024-04-15T15:16:26.884989+0000","last_fullsized":"2024-04-15T15:16:26.884989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885622+0000","last_change":"2024-04-15T15:16:25.879663+0000","last_active":"2024-04-15T15:16:26.885622+0000","last_peered":"2024-04-15T15:16:26.885622+0000","last_clean":"2024-04-15T15:16:26.885622+0000","last_became_active":"2024-04-15T15:16:25.879385+0000","last_became_peered":"2024-04-15T15:16:25.879385+0000","last_unstale":"2024-04-15T15:16:26.885622+0000","last_undegraded":"2024-04-15T15:16:26.885622+0000","last_fullsized":"2024-04-15T15:16:26.885622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888033+0000","last_change":"2024-04-15T15:16:25.886972+0000","last_active":"2024-04-15T15:16:25.888033+0000","last_peered":"2024-04-15T15:16:25.888033+0000","last_clean":"2024-04-15T15:16:25.888033+0000","last_became_active":"2024-04-15T15:16:25.886718+0000","last_became_peered":"2024-04-15T15:16:25.886718+0000","last_unstale":"2024-04-15T15:16:25.888033+0000","last_undegraded":"2024-04-15T15:16:25.888033+0000","last_fullsized":"2024-04-15T15:16:25.888033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885577+0000","last_change":"2024-04-15T15:16:24.886178+0000","last_active":"2024-04-15T15:16:26.885577+0000","last_peered":"2024-04-15T15:16:26.885577+0000","last_clean":"2024-04-15T15:16:26.885577+0000","last_became_active":"2024-04-15T15:16:24.885910+0000","last_became_peered":"2024-04-15T15:16:24.885910+0000","last_unstale":"2024-04-15T15:16:26.885577+0000","last_undegraded":"2024-04-15T15:16:26.885577+0000","last_fullsized":"2024-04-15T15:16:26.885577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903528+0000","last_change":"2024-04-15T15:16:25.902820+0000","last_active":"2024-04-15T15:16:25.903528+0000","last_peered":"2024-04-15T15:16:25.903528+0000","last_clean":"2024-04-15T15:16:25.903528+0000","last_became_active":"2024-04-15T15:16:25.902552+0000","last_became_peered":"2024-04-15T15:16:25.902552+0000","last_unstale":"2024-04-15T15:16:25.903528+0000","last_undegraded":"2024-04-15T15:16:25.903528+0000","last_fullsized":"2024-04-15T15:16:25.903528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898338+0000","last_change":"2024-04-15T15:16:24.886331+0000","last_active":"2024-04-15T15:16:25.898338+0000","last_peered":"2024-04-15T15:16:25.898338+0000","last_clean":"2024-04-15T15:16:25.898338+0000","last_became_active":"2024-04-15T15:16:24.886052+0000","last_became_peered":"2024-04-15T15:16:24.886052+0000","last_unstale":"2024-04-15T15:16:25.898338+0000","last_undegraded":"2024-04-15T15:16:25.898338+0000","last_fullsized":"2024-04-15T15:16:25.898338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.899883+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.899620+0000","last_became_peered":"2024-04-15T15:16:25.899620+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898694+0000","last_change":"2024-04-15T15:16:24.890646+0000","last_active":"2024-04-15T15:16:25.898694+0000","last_peered":"2024-04-15T15:16:25.898694+0000","last_clean":"2024-04-15T15:16:25.898694+0000","last_became_active":"2024-04-15T15:16:24.890362+0000","last_became_peered":"2024-04-15T15:16:24.890362+0000","last_unstale":"2024-04-15T15:16:25.898694+0000","last_undegraded":"2024-04-15T15:16:25.898694+0000","last_fullsized":"2024-04-15T15:16:25.898694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883847+0000","last_change":"2024-04-15T15:16:25.895423+0000","last_active":"2024-04-15T15:16:26.883847+0000","last_peered":"2024-04-15T15:16:26.883847+0000","last_clean":"2024-04-15T15:16:26.883847+0000","last_became_active":"2024-04-15T15:16:25.889905+0000","last_became_peered":"2024-04-15T15:16:25.889905+0000","last_unstale":"2024-04-15T15:16:26.883847+0000","last_undegraded":"2024-04-15T15:16:26.883847+0000","last_fullsized":"2024-04-15T15:16:26.883847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:50:15.175711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886111+0000","last_change":"2024-04-15T15:16:24.887352+0000","last_active":"2024-04-15T15:16:26.886111+0000","last_peered":"2024-04-15T15:16:26.886111+0000","last_clean":"2024-04-15T15:16:26.886111+0000","last_became_active":"2024-04-15T15:16:24.887054+0000","last_became_peered":"2024-04-15T15:16:24.887054+0000","last_unstale":"2024-04-15T15:16:26.886111+0000","last_undegraded":"2024-04-15T15:16:26.886111+0000","last_fullsized":"2024-04-15T15:16:26.886111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.882206+0000","last_change":"2024-04-15T15:16:25.881247+0000","last_active":"2024-04-15T15:16:25.882206+0000","last_peered":"2024-04-15T15:16:25.882206+0000","last_clean":"2024-04-15T15:16:25.882206+0000","last_became_active":"2024-04-15T15:16:25.880981+0000","last_became_peered":"2024-04-15T15:16:25.880981+0000","last_unstale":"2024-04-15T15:16:25.882206+0000","last_undegraded":"2024-04-15T15:16:25.882206+0000","last_fullsized":"2024-04-15T15:16:25.882206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903854+0000","last_change":"2024-04-15T15:16:24.891022+0000","last_active":"2024-04-15T15:16:25.903854+0000","last_peered":"2024-04-15T15:16:25.903854+0000","last_clean":"2024-04-15T15:16:25.903854+0000","last_became_active":"2024-04-15T15:16:24.890770+0000","last_became_peered":"2024-04-15T15:16:24.890770+0000","last_unstale":"2024-04-15T15:16:25.903854+0000","last_undegraded":"2024-04-15T15:16:25.903854+0000","last_fullsized":"2024-04-15T15:16:25.903854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901334+0000","last_change":"2024-04-15T15:16:25.893573+0000","last_active":"2024-04-15T15:16:25.901334+0000","last_peered":"2024-04-15T15:16:25.901334+0000","last_clean":"2024-04-15T15:16:25.901334+0000","last_became_active":"2024-04-15T15:16:25.893321+0000","last_became_peered":"2024-04-15T15:16:25.893321+0000","last_unstale":"2024-04-15T15:16:25.901334+0000","last_undegraded":"2024-04-15T15:16:25.901334+0000","last_fullsized":"2024-04-15T15:16:25.901334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885832+0000","last_change":"2024-04-15T15:16:25.904897+0000","last_active":"2024-04-15T15:16:26.885832+0000","last_peered":"2024-04-15T15:16:26.885832+0000","last_clean":"2024-04-15T15:16:26.885832+0000","last_became_active":"2024-04-15T15:16:25.904610+0000","last_became_peered":"2024-04-15T15:16:25.904610+0000","last_unstale":"2024-04-15T15:16:26.885832+0000","last_undegraded":"2024-04-15T15:16:26.885832+0000","last_fullsized":"2024-04-15T15:16:26.885832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876235+0000","last_change":"2024-04-15T15:16:24.891428+0000","last_active":"2024-04-15T15:16:25.876235+0000","last_peered":"2024-04-15T15:16:25.876235+0000","last_clean":"2024-04-15T15:16:25.876235+0000","last_became_active":"2024-04-15T15:16:24.891151+0000","last_became_peered":"2024-04-15T15:16:24.891151+0000","last_unstale":"2024-04-15T15:16:25.876235+0000","last_undegraded":"2024-04-15T15:16:25.876235+0000","last_fullsized":"2024-04-15T15:16:25.876235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:56:54.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883767+0000","last_change":"2024-04-15T15:16:25.898140+0000","last_active":"2024-04-15T15:16:26.883767+0000","last_peered":"2024-04-15T15:16:26.883767+0000","last_clean":"2024-04-15T15:16:26.883767+0000","last_became_active":"2024-04-15T15:16:25.897492+0000","last_became_peered":"2024-04-15T15:16:25.897492+0000","last_unstale":"2024-04-15T15:16:26.883767+0000","last_undegraded":"2024-04-15T15:16:26.883767+0000","last_fullsized":"2024-04-15T15:16:26.883767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884775+0000","last_change":"2024-04-15T15:16:24.884571+0000","last_active":"2024-04-15T15:16:26.884775+0000","last_peered":"2024-04-15T15:16:26.884775+0000","last_clean":"2024-04-15T15:16:26.884775+0000","last_became_active":"2024-04-15T15:16:24.884274+0000","last_became_peered":"2024-04-15T15:16:24.884274+0000","last_unstale":"2024-04-15T15:16:26.884775+0000","last_undegraded":"2024-04-15T15:16:26.884775+0000","last_fullsized":"2024-04-15T15:16:26.884775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.899098+0000","last_change":"2024-04-15T15:16:25.898346+0000","last_active":"2024-04-15T15:16:25.899098+0000","last_peered":"2024-04-15T15:16:25.899098+0000","last_clean":"2024-04-15T15:16:25.899098+0000","last_became_active":"2024-04-15T15:16:25.897145+0000","last_became_peered":"2024-04-15T15:16:25.897145+0000","last_unstale":"2024-04-15T15:16:25.899098+0000","last_undegraded":"2024-04-15T15:16:25.899098+0000","last_fullsized":"2024-04-15T15:16:25.899098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:54.775491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884135+0000","last_change":"2024-04-15T15:16:24.888936+0000","last_active":"2024-04-15T15:16:26.884135+0000","last_peered":"2024-04-15T15:16:26.884135+0000","last_clean":"2024-04-15T15:16:26.884135+0000","last_became_active":"2024-04-15T15:16:24.888050+0000","last_became_peered":"2024-04-15T15:16:24.888050+0000","last_unstale":"2024-04-15T15:16:26.884135+0000","last_undegraded":"2024-04-15T15:16:26.884135+0000","last_fullsized":"2024-04-15T15:16:26.884135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.886799+0000","last_change":"2024-04-15T15:16:24.887199+0000","last_active":"2024-04-15T15:16:25.886799+0000","last_peered":"2024-04-15T15:16:25.886799+0000","last_clean":"2024-04-15T15:16:25.886799+0000","last_became_active":"2024-04-15T15:16:24.886945+0000","last_became_peered":"2024-04-15T15:16:24.886945+0000","last_unstale":"2024-04-15T15:16:25.886799+0000","last_undegraded":"2024-04-15T15:16:25.886799+0000","last_fullsized":"2024-04-15T15:16:25.886799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.893867+0000","last_change":"2024-04-15T15:16:25.892203+0000","last_active":"2024-04-15T15:16:25.893867+0000","last_peered":"2024-04-15T15:16:25.893867+0000","last_clean":"2024-04-15T15:16:25.893867+0000","last_became_active":"2024-04-15T15:16:25.891899+0000","last_became_peered":"2024-04-15T15:16:25.891899+0000","last_unstale":"2024-04-15T15:16:25.893867+0000","last_undegraded":"2024-04-15T15:16:25.893867+0000","last_fullsized":"2024-04-15T15:16:25.893867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904566+0000","last_change":"2024-04-15T15:16:24.890265+0000","last_active":"2024-04-15T15:16:25.904566+0000","last_peered":"2024-04-15T15:16:25.904566+0000","last_clean":"2024-04-15T15:16:25.904566+0000","last_became_active":"2024-04-15T15:16:24.889963+0000","last_became_peered":"2024-04-15T15:16:24.889963+0000","last_unstale":"2024-04-15T15:16:25.904566+0000","last_undegraded":"2024-04-15T15:16:25.904566+0000","last_fullsized":"2024-04-15T15:16:25.904566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886503+0000","last_change":"2024-04-15T15:16:25.893921+0000","last_active":"2024-04-15T15:16:26.886503+0000","last_peered":"2024-04-15T15:16:26.886503+0000","last_clean":"2024-04-15T15:16:26.886503+0000","last_became_active":"2024-04-15T15:16:25.892154+0000","last_became_peered":"2024-04-15T15:16:25.892154+0000","last_unstale":"2024-04-15T15:16:26.886503+0000","last_undegraded":"2024-04-15T15:16:26.886503+0000","last_fullsized":"2024-04-15T15:16:26.886503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884185+0000","last_change":"2024-04-15T15:16:24.889313+0000","last_active":"2024-04-15T15:16:26.884185+0000","last_peered":"2024-04-15T15:16:26.884185+0000","last_clean":"2024-04-15T15:16:26.884185+0000","last_became_active":"2024-04-15T15:16:24.889037+0000","last_became_peered":"2024-04-15T15:16:24.889037+0000","last_unstale":"2024-04-15T15:16:26.884185+0000","last_undegraded":"2024-04-15T15:16:26.884185+0000","last_fullsized":"2024-04-15T15:16:26.884185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:58.773260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.889667+0000","last_change":"2024-04-15T15:16:25.888305+0000","last_active":"2024-04-15T15:16:25.889667+0000","last_peered":"2024-04-15T15:16:25.889667+0000","last_clean":"2024-04-15T15:16:25.889667+0000","last_became_active":"2024-04-15T15:16:25.888053+0000","last_became_peered":"2024-04-15T15:16:25.888053+0000","last_unstale":"2024-04-15T15:16:25.889667+0000","last_undegraded":"2024-04-15T15:16:25.889667+0000","last_fullsized":"2024-04-15T15:16:25.889667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884136+0000","last_change":"2024-04-15T15:16:24.882824+0000","last_active":"2024-04-15T15:16:26.884136+0000","last_peered":"2024-04-15T15:16:26.884136+0000","last_clean":"2024-04-15T15:16:26.884136+0000","last_became_active":"2024-04-15T15:16:24.882560+0000","last_became_peered":"2024-04-15T15:16:24.882560+0000","last_unstale":"2024-04-15T15:16:26.884136+0000","last_undegraded":"2024-04-15T15:16:26.884136+0000","last_fullsized":"2024-04-15T15:16:26.884136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:08.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883781+0000","last_change":"2024-04-15T15:16:25.883512+0000","last_active":"2024-04-15T15:16:26.883781+0000","last_peered":"2024-04-15T15:16:26.883781+0000","last_clean":"2024-04-15T15:16:26.883781+0000","last_became_active":"2024-04-15T15:16:25.883253+0000","last_became_peered":"2024-04-15T15:16:25.883253+0000","last_unstale":"2024-04-15T15:16:26.883781+0000","last_undegraded":"2024-04-15T15:16:26.883781+0000","last_fullsized":"2024-04-15T15:16:26.883781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.877572+0000","last_change":"2024-04-15T15:16:24.891194+0000","last_active":"2024-04-15T15:16:25.877572+0000","last_peered":"2024-04-15T15:16:25.877572+0000","last_clean":"2024-04-15T15:16:25.877572+0000","last_became_active":"2024-04-15T15:16:24.890944+0000","last_became_peered":"2024-04-15T15:16:24.890944+0000","last_unstale":"2024-04-15T15:16:25.877572+0000","last_undegraded":"2024-04-15T15:16:25.877572+0000","last_fullsized":"2024-04-15T15:16:25.877572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885147+0000","last_change":"2024-04-15T15:16:24.886029+0000","last_active":"2024-04-15T15:16:26.885147+0000","last_peered":"2024-04-15T15:16:26.885147+0000","last_clean":"2024-04-15T15:16:26.885147+0000","last_became_active":"2024-04-15T15:16:24.885730+0000","last_became_peered":"2024-04-15T15:16:24.885730+0000","last_unstale":"2024-04-15T15:16:26.885147+0000","last_undegraded":"2024-04-15T15:16:26.885147+0000","last_fullsized":"2024-04-15T15:16:26.885147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884020+0000","last_change":"2024-04-15T15:16:25.882916+0000","last_active":"2024-04-15T15:16:26.884020+0000","last_peered":"2024-04-15T15:16:26.884020+0000","last_clean":"2024-04-15T15:16:26.884020+0000","last_became_active":"2024-04-15T15:16:25.882630+0000","last_became_peered":"2024-04-15T15:16:25.882630+0000","last_unstale":"2024-04-15T15:16:26.884020+0000","last_undegraded":"2024-04-15T15:16:26.884020+0000","last_fullsized":"2024-04-15T15:16:26.884020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:24.310942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884117+0000","last_change":"2024-04-15T15:16:25.890929+0000","last_active":"2024-04-15T15:16:26.884117+0000","last_peered":"2024-04-15T15:16:26.884117+0000","last_clean":"2024-04-15T15:16:26.884117+0000","last_became_active":"2024-04-15T15:16:25.890256+0000","last_became_peered":"2024-04-15T15:16:25.890256+0000","last_unstale":"2024-04-15T15:16:26.884117+0000","last_undegraded":"2024-04-15T15:16:26.884117+0000","last_fullsized":"2024-04-15T15:16:26.884117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:11.834293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906758+0000","last_change":"2024-04-15T15:16:24.893507+0000","last_active":"2024-04-15T15:16:25.906758+0000","last_peered":"2024-04-15T15:16:25.906758+0000","last_clean":"2024-04-15T15:16:25.906758+0000","last_became_active":"2024-04-15T15:16:24.893262+0000","last_became_peered":"2024-04-15T15:16:24.893262+0000","last_unstale":"2024-04-15T15:16:25.906758+0000","last_undegraded":"2024-04-15T15:16:25.906758+0000","last_fullsized":"2024-04-15T15:16:25.906758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:15.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911705+0000","last_change":"2024-04-15T15:16:25.904384+0000","last_active":"2024-04-15T15:16:25.911705+0000","last_peered":"2024-04-15T15:16:25.911705+0000","last_clean":"2024-04-15T15:16:25.911705+0000","last_became_active":"2024-04-15T15:16:25.904095+0000","last_became_peered":"2024-04-15T15:16:25.904095+0000","last_unstale":"2024-04-15T15:16:25.911705+0000","last_undegraded":"2024-04-15T15:16:25.911705+0000","last_fullsized":"2024-04-15T15:16:25.911705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886583+0000","last_change":"2024-04-15T15:16:24.891423+0000","last_active":"2024-04-15T15:16:26.886583+0000","last_peered":"2024-04-15T15:16:26.886583+0000","last_clean":"2024-04-15T15:16:26.886583+0000","last_became_active":"2024-04-15T15:16:24.891048+0000","last_became_peered":"2024-04-15T15:16:24.891048+0000","last_unstale":"2024-04-15T15:16:26.886583+0000","last_undegraded":"2024-04-15T15:16:26.886583+0000","last_fullsized":"2024-04-15T15:16:26.886583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:47:10.367180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.894040+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.893781+0000","last_became_peered":"2024-04-15T15:16:25.893781+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.897493+0000","last_change":"2024-04-15T15:16:24.889598+0000","last_active":"2024-04-15T15:16:25.897493+0000","last_peered":"2024-04-15T15:16:25.897493+0000","last_clean":"2024-04-15T15:16:25.897493+0000","last_became_active":"2024-04-15T15:16:24.889298+0000","last_became_peered":"2024-04-15T15:16:24.889298+0000","last_unstale":"2024-04-15T15:16:25.897493+0000","last_undegraded":"2024-04-15T15:16:25.897493+0000","last_fullsized":"2024-04-15T15:16:25.897493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:54:18.385006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891485+0000","last_change":"2024-04-15T15:16:25.884522+0000","last_active":"2024-04-15T15:16:25.891485+0000","last_peered":"2024-04-15T15:16:25.891485+0000","last_clean":"2024-04-15T15:16:25.891485+0000","last_became_active":"2024-04-15T15:16:25.883126+0000","last_became_peered":"2024-04-15T15:16:25.883126+0000","last_unstale":"2024-04-15T15:16:25.891485+0000","last_undegraded":"2024-04-15T15:16:25.891485+0000","last_fullsized":"2024-04-15T15:16:25.891485+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:03.969456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876163+0000","last_change":"2024-04-15T15:16:24.890031+0000","last_active":"2024-04-15T15:16:25.876163+0000","last_peered":"2024-04-15T15:16:25.876163+0000","last_clean":"2024-04-15T15:16:25.876163+0000","last_became_active":"2024-04-15T15:16:24.889203+0000","last_became_peered":"2024-04-15T15:16:24.889203+0000","last_unstale":"2024-04-15T15:16:25.876163+0000","last_undegraded":"2024-04-15T15:16:25.876163+0000","last_fullsized":"2024-04-15T15:16:25.876163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:18.919033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891401+0000","last_change":"2024-04-15T15:16:24.877420+0000","last_active":"2024-04-15T15:16:25.891401+0000","last_peered":"2024-04-15T15:16:25.891401+0000","last_clean":"2024-04-15T15:16:25.891401+0000","last_became_active":"2024-04-15T15:16:24.877149+0000","last_became_peered":"2024-04-15T15:16:24.877149+0000","last_unstale":"2024-04-15T15:16:25.891401+0000","last_undegraded":"2024-04-15T15:16:25.891401+0000","last_fullsized":"2024-04-15T15:16:25.891401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:48.237807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.890064+0000","last_change":"2024-04-15T15:16:25.887625+0000","last_active":"2024-04-15T15:16:25.890064+0000","last_peered":"2024-04-15T15:16:25.890064+0000","last_clean":"2024-04-15T15:16:25.890064+0000","last_became_active":"2024-04-15T15:16:25.887362+0000","last_became_peered":"2024-04-15T15:16:25.887362+0000","last_unstale":"2024-04-15T15:16:25.890064+0000","last_undegraded":"2024-04-15T15:16:25.890064+0000","last_fullsized":"2024-04-15T15:16:25.890064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872610+0000","last_change":"2024-04-15T15:16:24.884150+0000","last_active":"2024-04-15T15:16:25.872610+0000","last_peered":"2024-04-15T15:16:25.872610+0000","last_clean":"2024-04-15T15:16:25.872610+0000","last_became_active":"2024-04-15T15:16:24.883908+0000","last_became_peered":"2024-04-15T15:16:24.883908+0000","last_unstale":"2024-04-15T15:16:25.872610+0000","last_undegraded":"2024-04-15T15:16:25.872610+0000","last_fullsized":"2024-04-15T15:16:25.872610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:58:35.147306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903908+0000","last_change":"2024-04-15T15:16:25.897701+0000","last_active":"2024-04-15T15:16:25.903908+0000","last_peered":"2024-04-15T15:16:25.903908+0000","last_clean":"2024-04-15T15:16:25.903908+0000","last_became_active":"2024-04-15T15:16:25.896891+0000","last_became_peered":"2024-04-15T15:16:25.896891+0000","last_unstale":"2024-04-15T15:16:25.903908+0000","last_undegraded":"2024-04-15T15:16:25.903908+0000","last_fullsized":"2024-04-15T15:16:25.903908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884030+0000","last_change":"2024-04-15T15:16:25.899465+0000","last_active":"2024-04-15T15:16:26.884030+0000","last_peered":"2024-04-15T15:16:26.884030+0000","last_clean":"2024-04-15T15:16:26.884030+0000","last_became_active":"2024-04-15T15:16:25.899205+0000","last_became_peered":"2024-04-15T15:16:25.899205+0000","last_unstale":"2024-04-15T15:16:26.884030+0000","last_undegraded":"2024-04-15T15:16:26.884030+0000","last_fullsized":"2024-04-15T15:16:26.884030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:42:16.777997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872575+0000","last_change":"2024-04-15T15:16:24.882714+0000","last_active":"2024-04-15T15:16:25.872575+0000","last_peered":"2024-04-15T15:16:25.872575+0000","last_clean":"2024-04-15T15:16:25.872575+0000","last_became_active":"2024-04-15T15:16:24.882460+0000","last_became_peered":"2024-04-15T15:16:24.882460+0000","last_unstale":"2024-04-15T15:16:25.872575+0000","last_undegraded":"2024-04-15T15:16:25.872575+0000","last_fullsized":"2024-04-15T15:16:25.872575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:11.251738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898832+0000","last_change":"2024-04-15T15:16:25.897148+0000","last_active":"2024-04-15T15:16:25.898832+0000","last_peered":"2024-04-15T15:16:25.898832+0000","last_clean":"2024-04-15T15:16:25.898832+0000","last_became_active":"2024-04-15T15:16:25.896866+0000","last_became_peered":"2024-04-15T15:16:25.896866+0000","last_unstale":"2024-04-15T15:16:25.898832+0000","last_undegraded":"2024-04-15T15:16:25.898832+0000","last_fullsized":"2024-04-15T15:16:25.898832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902029+0000","last_change":"2024-04-15T15:16:24.892956+0000","last_active":"2024-04-15T15:16:25.902029+0000","last_peered":"2024-04-15T15:16:25.902029+0000","last_clean":"2024-04-15T15:16:25.902029+0000","last_became_active":"2024-04-15T15:16:24.892664+0000","last_became_peered":"2024-04-15T15:16:24.892664+0000","last_unstale":"2024-04-15T15:16:25.902029+0000","last_undegraded":"2024-04-15T15:16:25.902029+0000","last_fullsized":"2024-04-15T15:16:25.902029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:56.013415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.873000+0000","last_change":"2024-04-15T15:16:24.890594+0000","last_active":"2024-04-15T15:16:25.873000+0000","last_peered":"2024-04-15T15:16:25.873000+0000","last_clean":"2024-04-15T15:16:25.873000+0000","last_became_active":"2024-04-15T15:16:24.890305+0000","last_became_peered":"2024-04-15T15:16:24.890305+0000","last_unstale":"2024-04-15T15:16:25.873000+0000","last_undegraded":"2024-04-15T15:16:25.873000+0000","last_fullsized":"2024-04-15T15:16:25.873000+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:34.555529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884060+0000","last_change":"2024-04-15T15:16:25.893319+0000","last_active":"2024-04-15T15:16:26.884060+0000","last_peered":"2024-04-15T15:16:26.884060+0000","last_clean":"2024-04-15T15:16:26.884060+0000","last_became_active":"2024-04-15T15:16:25.892794+0000","last_became_peered":"2024-04-15T15:16:25.892794+0000","last_unstale":"2024-04-15T15:16:26.884060+0000","last_undegraded":"2024-04-15T15:16:26.884060+0000","last_fullsized":"2024-04-15T15:16:26.884060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:42.664115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876776+0000","last_change":"2024-04-15T15:16:24.884802+0000","last_active":"2024-04-15T15:16:25.876776+0000","last_peered":"2024-04-15T15:16:25.876776+0000","last_clean":"2024-04-15T15:16:25.876776+0000","last_became_active":"2024-04-15T15:16:24.884519+0000","last_became_peered":"2024-04-15T15:16:24.884519+0000","last_unstale":"2024-04-15T15:16:25.876776+0000","last_undegraded":"2024-04-15T15:16:25.876776+0000","last_fullsized":"2024-04-15T15:16:25.876776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888791+0000","last_change":"2024-04-15T15:16:25.883444+0000","last_active":"2024-04-15T15:16:25.888791+0000","last_peered":"2024-04-15T15:16:25.888791+0000","last_clean":"2024-04-15T15:16:25.888791+0000","last_became_active":"2024-04-15T15:16:25.883191+0000","last_became_peered":"2024-04-15T15:16:25.883191+0000","last_unstale":"2024-04-15T15:16:25.888791+0000","last_undegraded":"2024-04-15T15:16:25.888791+0000","last_fullsized":"2024-04-15T15:16:25.888791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.875590+0000","last_change":"2024-04-15T15:16:24.886148+0000","last_active":"2024-04-15T15:16:25.875590+0000","last_peered":"2024-04-15T15:16:25.875590+0000","last_clean":"2024-04-15T15:16:25.875590+0000","last_became_active":"2024-04-15T15:16:24.885865+0000","last_became_peered":"2024-04-15T15:16:24.885865+0000","last_unstale":"2024-04-15T15:16:25.875590+0000","last_undegraded":"2024-04-15T15:16:25.875590+0000","last_fullsized":"2024-04-15T15:16:25.875590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888002+0000","last_change":"2024-04-15T15:16:25.886178+0000","last_active":"2024-04-15T15:16:25.888002+0000","last_peered":"2024-04-15T15:16:25.888002+0000","last_clean":"2024-04-15T15:16:25.888002+0000","last_became_active":"2024-04-15T15:16:25.885915+0000","last_became_peered":"2024-04-15T15:16:25.885915+0000","last_unstale":"2024-04-15T15:16:25.888002+0000","last_undegraded":"2024-04-15T15:16:25.888002+0000","last_fullsized":"2024-04-15T15:16:25.888002+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:33.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885719+0000","last_change":"2024-04-15T15:16:24.883958+0000","last_active":"2024-04-15T15:16:26.885719+0000","last_peered":"2024-04-15T15:16:26.885719+0000","last_clean":"2024-04-15T15:16:26.885719+0000","last_became_active":"2024-04-15T15:16:24.883528+0000","last_became_peered":"2024-04-15T15:16:24.883528+0000","last_unstale":"2024-04-15T15:16:26.885719+0000","last_undegraded":"2024-04-15T15:16:26.885719+0000","last_fullsized":"2024-04-15T15:16:26.885719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885492+0000","last_change":"2024-04-15T15:16:25.889861+0000","last_active":"2024-04-15T15:16:26.885492+0000","last_peered":"2024-04-15T15:16:26.885492+0000","last_clean":"2024-04-15T15:16:26.885492+0000","last_became_active":"2024-04-15T15:16:25.889591+0000","last_became_peered":"2024-04-15T15:16:25.889591+0000","last_unstale":"2024-04-15T15:16:26.885492+0000","last_undegraded":"2024-04-15T15:16:26.885492+0000","last_fullsized":"2024-04-15T15:16:26.885492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884867+0000","last_change":"2024-04-15T15:16:24.891429+0000","last_active":"2024-04-15T15:16:26.884867+0000","last_peered":"2024-04-15T15:16:26.884867+0000","last_clean":"2024-04-15T15:16:26.884867+0000","last_became_active":"2024-04-15T15:16:24.891111+0000","last_became_peered":"2024-04-15T15:16:24.891111+0000","last_unstale":"2024-04-15T15:16:26.884867+0000","last_undegraded":"2024-04-15T15:16:26.884867+0000","last_fullsized":"2024-04-15T15:16:26.884867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:24.865017+0000","last_change":"2024-04-15T15:16:24.865017+0000","last_active":"2024-04-15T15:16:24.865017+0000","last_peered":"2024-04-15T15:16:24.865017+0000","last_clean":"2024-04-15T15:16:24.865017+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:24.865017+0000","last_undegraded":"2024-04-15T15:16:24.865017+0000","last_fullsized":"2024-04-15T15:16:24.865017+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883631+0000","last_change":"2024-04-15T15:16:24.887452+0000","last_active":"2024-04-15T15:16:26.883631+0000","last_peered":"2024-04-15T15:16:26.883631+0000","last_clean":"2024-04-15T15:16:26.883631+0000","last_became_active":"2024-04-15T15:16:24.887136+0000","last_became_peered":"2024-04-15T15:16:24.887136+0000","last_unstale":"2024-04-15T15:16:26.883631+0000","last_undegraded":"2024-04-15T15:16:26.883631+0000","last_fullsized":"2024-04-15T15:16:26.883631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:24.817169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906655+0000","last_change":"2024-04-15T15:16:25.901054+0000","last_active":"2024-04-15T15:16:25.906655+0000","last_peered":"2024-04-15T15:16:25.906655+0000","last_clean":"2024-04-15T15:16:25.906655+0000","last_became_active":"2024-04-15T15:16:25.900784+0000","last_became_peered":"2024-04-15T15:16:25.900784+0000","last_unstale":"2024-04-15T15:16:25.906655+0000","last_undegraded":"2024-04-15T15:16:25.906655+0000","last_fullsized":"2024-04-15T15:16:25.906655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-15T15:16:23.862718+0000","last_change":"2024-04-15T15:16:23.862718+0000","last_active":"2024-04-15T15:16:23.862718+0000","last_peered":"2024-04-15T15:16:23.862718+0000","last_clean":"2024-04-15T15:16:23.862718+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-15T15:16:23.862718+0000","last_undegraded":"2024-04-15T15:16:23.862718+0000","last_fullsized":"2024-04-15T15:16:23.862718+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":195,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-04-15T15:16:20.700430+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:16:20.700430+0000","last_peered":"2024-04-15T15:16:20.700430+0000","last_clean":"2024-04-15T15:16:20.700430+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:16:20.700430+0000","last_undegraded":"2024-04-15T15:16:20.700430+0000","last_fullsized":"2024-04-15T15:16:20.700430+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":108,"acting":108,"num_store_stats":7},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":9,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":56,"acting":56,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27996,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343844,"statfs":{"total":96636764160,"available":96608096256,"internally_reserved":0,"allocated":995328,"data_stored":758052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":189373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":778129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-15T15:16:29.579 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:23.862718+0000', 'last_change': '2024-04-15T15:16:23.862718+0000', 'last_active': '2024-04-15T15:16:23.862718+0000', 'last_peered': '2024-04-15T15:16:23.862718+0000', 'last_clean': '2024-04-15T15:16:23.862718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:23.862718+0000', 'last_undegraded': '2024-04-15T15:16:23.862718+0000', 'last_fullsized': '2024-04-15T15:16:23.862718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:23.862718+0000', 'last_change': '2024-04-15T15:16:23.862718+0000', 'last_active': '2024-04-15T15:16:23.862718+0000', 'last_peered': '2024-04-15T15:16:23.862718+0000', 'last_clean': '2024-04-15T15:16:23.862718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:23.862718+0000', 'last_undegraded': '2024-04-15T15:16:23.862718+0000', 'last_fullsized': '2024-04-15T15:16:23.862718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.580 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:23.862718+0000', 'last_change': '2024-04-15T15:16:23.862718+0000', 'last_active': '2024-04-15T15:16:23.862718+0000', 'last_peered': '2024-04-15T15:16:23.862718+0000', 'last_clean': '2024-04-15T15:16:23.862718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:23.862718+0000', 'last_undegraded': '2024-04-15T15:16:23.862718+0000', 'last_fullsized': '2024-04-15T15:16:23.862718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:24.865017+0000', 'last_change': '2024-04-15T15:16:24.865017+0000', 'last_active': '2024-04-15T15:16:24.865017+0000', 'last_peered': '2024-04-15T15:16:24.865017+0000', 'last_clean': '2024-04-15T15:16:24.865017+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:24.865017+0000', 'last_undegraded': '2024-04-15T15:16:24.865017+0000', 'last_fullsized': '2024-04-15T15:16:24.865017+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:24.865017+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-04-15T15:16:29.581 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-15T15:16:23.862718+0000', 'last_change': '2024-04-15T15:16:23.862718+0000', 'last_active': '2024-04-15T15:16:23.862718+0000', 'last_peered': '2024-04-15T15:16:23.862718+0000', 'last_clean': '2024-04-15T15:16:23.862718+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-15T15:16:23.862718+0000', 'last_undegraded': '2024-04-15T15:16:23.862718+0000', 'last_fullsized': '2024-04-15T15:16:23.862718+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'last_clean_scrub_stamp': '2024-04-15T15:16:23.862718+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-15T15:16:29.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-15T15:16:29.719 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.715+0000 7f29fb4ba640 1 Processor -- start 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 -- start start 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f29f415f750 0x7f29f40d2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f29f415fbe0 0x7f29f40d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f40cf600 con 0x7f29f40d0de0 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f29f40cf8e0 con 0x7f29f415fbe0 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29fb4ba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f29f40cfbc0 con 0x7f29f415f750 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29f9a57640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f29f415fbe0 0x7f29f40d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29f9a57640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f29f415fbe0 0x7f29f40d26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49816/0 (socket says 172.21.15.70:49816) 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.716+0000 7f29f9a57640 1 -- 172.21.15.70:0/3063230053 learned_addr learned my addr 172.21.15.70:0/3063230053 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f8a55640 1 --2- 172.21.15.70:0/3063230053 >> v2:172.21.15.96:3300/0 conn(0x7f29f415f750 0x7f29f40d2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f9a57640 1 -- 172.21.15.70:0/3063230053 >> v2:172.21.15.96:3300/0 conn(0x7f29f415f750 msgr2=0x7f29f40d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f9a57640 1 --2- 172.21.15.70:0/3063230053 >> v2:172.21.15.96:3300/0 conn(0x7f29f415f750 0x7f29f40d2170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.720 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f9a57640 1 -- 172.21.15.70:0/3063230053 >> v1:172.21.15.96:6789/0 conn(0x7f29f40d0de0 legacy=0x7f29f40d11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f9a57640 1 -- 172.21.15.70:0/3063230053 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f40cfea0 con 0x7f29f415fbe0 2024-04-15T15:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.717+0000 7f29f9a57640 1 --2- 172.21.15.70:0/3063230053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f415fbe0 0x7f29f40d26b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f29f408da60 tx=0x7f29ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3633f7c942d277a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29ebfff640 1 -- 172.21.15.70:0/3063230053 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec035020 con 0x7f29f415fbe0 2024-04-15T15:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29ebfff640 1 -- 172.21.15.70:0/3063230053 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29ec031870 con 0x7f29f415fbe0 2024-04-15T15:16:29.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29ebfff640 1 -- 172.21.15.70:0/3063230053 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29ec031c40 con 0x7f29f415fbe0 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 -- 172.21.15.70:0/3063230053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f415fbe0 msgr2=0x7f29f40d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/3063230053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f415fbe0 0x7f29f40d26b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f29f408da60 tx=0x7f29ec02b910 comp rx=0 tx=0).stop 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 -- 172.21.15.70:0/3063230053 shutdown_connections 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/3063230053 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f415fbe0 0x7f29f40d26b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/3063230053 >> v2:172.21.15.96:3300/0 conn(0x7f29f415f750 0x7f29f40d2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 -- 172.21.15.70:0/3063230053 >> 172.21.15.70:0/3063230053 conn(0x7f29f408bf60 msgr2=0x7f29f40d45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 -- 172.21.15.70:0/3063230053 shutdown_connections 2024-04-15T15:16:29.724 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.718+0000 7f29fb4ba640 1 -- 172.21.15.70:0/3063230053 wait complete. 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.719+0000 7f29fb4ba640 1 Processor -- start 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 -- start start 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f29f415f750 0x7f29f41f84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29f415fbe0 0x7f29f415c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f29f415ebd0 con 0x7f29f40d0de0 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f29f415ea50 con 0x7f29f415fbe0 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29fb4ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f29f40cf1b0 con 0x7f29f415f750 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29f9256640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29f9256640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49832/0 (socket says 172.21.15.70:49832) 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.720+0000 7f29f9256640 1 -- 172.21.15.70:0/1810146537 learned_addr learned my addr 172.21.15.70:0/1810146537 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:29.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9a57640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29f415fbe0 0x7f29f415c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f8a55640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f29f415f750 0x7f29f41f84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 -- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f29f415f750 msgr2=0x7f29f41f84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f29f415f750 0x7f29f41f84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 -- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29f415fbe0 msgr2=0x7f29f415c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29f415fbe0 0x7f29f415c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 -- 172.21.15.70:0/1810146537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f40c87b0 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29f9256640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c7f0 tx=0x7f29f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f0016020 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29f000aa50 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29f40c8a90 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f000ae40 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.721+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29f415ced0 con 0x7f29f40d0de0 2024-04-15T15:16:29.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.722+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29f40c8fb0 con 0x7f29f40d0de0 2024-04-15T15:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.723+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f29f0019020 con 0x7f29f40d0de0 2024-04-15T15:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.723+0000 7f29e9ffb640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 0x7f29d806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.723+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f29f008b990 con 0x7f29f40d0de0 2024-04-15T15:16:29.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.723+0000 7f29f8a55640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 0x7f29d806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:29.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.724+0000 7f29f8a55640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 0x7f29d806edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29f41f7a10 tx=0x7f29e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:29.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.727+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f29f0020070 con 0x7f29f40d0de0 2024-04-15T15:16:29.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.975+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f29f40ca080 con 0x7f29f40d0de0 2024-04-15T15:16:29.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.975+0000 7f29e9ffb640 1 -- 172.21.15.70:0/1810146537 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1250 (secure 0 0 0) 0x7f29f0059cc0 con 0x7f29f40d0de0 2024-04-15T15:16:29.977 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:29.977 INFO:teuthology.orchestra.run.smithi070.stdout:{"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":104,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1713194097,"num_in_osds":8,"osd_in_since":1713194088,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":83},{"state_name":"unknown","count":14}],"num_pgs":97,"num_pools":3,"num_objects":19,"data_bytes":592778,"bytes_used":243105792,"bytes_avail":772851007488,"bytes_total":773094113280,"unknown_pgs_ratio":0.14432989060878754,"write_bytes_sec":3397,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":4907}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-15T15:16:06.081886+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-15T15:16:29.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 msgr2=0x7f29d806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 0x7f29d806edb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f29f41f7a10 tx=0x7f29e4008040 comp rx=0 tx=0).stop 2024-04-15T15:16:29.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 msgr2=0x7f29f40c80d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:29.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f29f000c7f0 tx=0x7f29f000ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 shutdown_connections 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f29f415fbe0 0x7f29f415c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f29f415f750 0x7f29f41f84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f29d806c8f0 0x7f29d806edb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 --2- 172.21.15.70:0/1810146537 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f29f40d0de0 0x7f29f40c80d0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 >> 172.21.15.70:0/1810146537 conn(0x7f29f408bf60 msgr2=0x7f29f40d45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 shutdown_connections 2024-04-15T15:16:29.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:29.977+0000 7f29fb4ba640 1 -- 172.21.15.70:0/1810146537 wait complete. 2024-04-15T15:16:29.990 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-15T15:16:29.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:30.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.091+0000 7f975958f640 1 Processor -- start 2024-04-15T15:16:30.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.091+0000 7f975958f640 1 -- start start 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f975958f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f97540b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f975958f640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f97540d2360 0x7f97540d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f975958f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97540b47a0 con 0x7f975415eae0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f975958f640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f97540b53b0 con 0x7f97540d2360 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f975958f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f97540b4a80 con 0x7f975415e700 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97537fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f975415eae0 0x7f97540cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:38884/0 (socket says 172.21.15.70:38884) 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97537fe640 1 -- 172.21.15.70:0/1499408843 learned_addr learned my addr 172.21.15.70:0/1499408843 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f974bfff640 1 --2- 172.21.15.70:0/1499408843 >> v2:172.21.15.96:3300/0 conn(0x7f97540d2360 0x7f97540d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f9752ffd640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f97540b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 348576580 0 0) 0x7f97540b47a0 con 0x7f975415eae0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9740003520 con 0x7f975415eae0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2637354159 0 0) 0x7f9740003520 con 0x7f975415eae0 2024-04-15T15:16:30.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 msgr2=0x7f97540b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f97540b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 >> v2:172.21.15.96:3300/0 conn(0x7f97540d2360 msgr2=0x7f97540d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 --2- 172.21.15.70:0/1499408843 >> v2:172.21.15.96:3300/0 conn(0x7f97540d2360 0x7f97540d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97540b4d60 con 0x7f975415eae0 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.092+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f974c0019f0 con 0x7f975415eae0 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 >> v1:172.21.15.70:6789/0 conn(0x7f975415eae0 legacy=0x7f97540cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400039b0 0x7f9740003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f9740004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9740004d40 0x7f97400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97540b53b0 con 0x7f9740004d40 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97540b4a80 con 0x7f97400039b0 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f974000b9c0 con 0x7f975415e700 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9740004d40 0x7f97400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f9752ffd640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400039b0 0x7f9740003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f974bfff640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f9740004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 -- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 msgr2=0x7f9740004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f9740004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 -- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400039b0 msgr2=0x7f9740003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f97400039b0 0x7f9740003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f97537fe640 1 -- 172.21.15.70:0/1499408843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9740003520 con 0x7f9740004d40 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.093+0000 7f974bfff640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f975415e700 0x7f9740004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f97537fe640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9740004d40 0x7f97400051c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f974c00a880 tx=0x7f9740005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2ad609fbe95962 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c00a4a0 con 0x7f9740004d40 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974c034650 con 0x7f9740004d40 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f97527fc640 1 -- 172.21.15.70:0/1499408843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c034a40 con 0x7f9740004d40 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f975958f640 1 -- 172.21.15.70:0/1499408843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9740004d40 msgr2=0x7f97400051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f975958f640 1 --2- 172.21.15.70:0/1499408843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9740004d40 0x7f97400051c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f974c00a880 tx=0x7f9740005830 comp rx=0 tx=0).stop 2024-04-15T15:16:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f97537fe640 1 -- 172.21.15.70:0/1499408843 reap_dead start 2024-04-15T15:16:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f975958f640 1 -- 172.21.15.70:0/1499408843 shutdown_connections 2024-04-15T15:16:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f975958f640 1 -- 172.21.15.70:0/1499408843 >> 172.21.15.70:0/1499408843 conn(0x7f975408bf60 msgr2=0x7f97540d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.094+0000 7f975958f640 1 -- 172.21.15.70:0/1499408843 shutdown_connections 2024-04-15T15:16:30.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.095+0000 7f975958f640 1 -- 172.21.15.70:0/1499408843 wait complete. 2024-04-15T15:16:30.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 Processor -- start 2024-04-15T15:16:30.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 -- start start 2024-04-15T15:16:30.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97540b7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97540c99b0 0x7f975414dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97540d1d40 con 0x7f975414e150 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f97540b4450 con 0x7f9740004d40 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f975958f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f97540d1ec0 con 0x7f97540c99b0 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.096+0000 7f97537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49850/0 (socket says 172.21.15.70:49850) 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 -- 172.21.15.70:0/727186249 learned_addr learned my addr 172.21.15.70:0/727186249 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f9752ffd640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97540b7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f974bfff640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97540c99b0 0x7f975414dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 -- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97540c99b0 msgr2=0x7f975414dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97540c99b0 0x7f975414dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 -- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 msgr2=0x7f97540b7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97540b7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 -- 172.21.15.70:0/727186249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97540c9eb0 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f97537fe640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f974c00a9a0 tx=0x7f97540b5320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c001ff0 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f974c044070 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97540ca190 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c034c00 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.097+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97540ca750 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.098+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97540689e0 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.099+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f974c049460 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.099+0000 7f974b7fe640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 0x7f973406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.099+0000 7f9752ffd640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 0x7f973406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.099+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f974c0ae200 con 0x7f975414e150 2024-04-15T15:16:30.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.100+0000 7f9752ffd640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 0x7f973406edb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f97540a2410 tx=0x7f9744008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:30.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.104+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f974c07c410 con 0x7f975414e150 2024-04-15T15:16:30.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.274+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97540ba500 con 0x7f973406c8f0 2024-04-15T15:16:30.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.280+0000 7f974b7fe640 1 -- 172.21.15.70:0/727186249 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251925 (secure 0 0 0) 0x7f97540ba500 con 0x7f973406c8f0 2024-04-15T15:16:30.285 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:30.287 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:16:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 msgr2=0x7f973406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 0x7f973406edb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f97540a2410 tx=0x7f9744008040 comp rx=0 tx=0).stop 2024-04-15T15:16:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 msgr2=0x7f975415a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f974c00a9a0 tx=0x7f97540b5320 comp rx=0 tx=0).stop 2024-04-15T15:16:30.288 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 shutdown_connections 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f973406c8f0 0x7f973406edb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f975414e150 0x7f975415a550 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97540c99b0 0x7f975414dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 --2- 172.21.15.70:0/727186249 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9740004d40 0x7f97540b7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 >> 172.21.15.70:0/727186249 conn(0x7f975408bf60 msgr2=0x7f975415f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 shutdown_connections 2024-04-15T15:16:30.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:30.283+0000 7f975958f640 1 -- 172.21.15.70:0/727186249 wait complete. 2024-04-15T15:16:30.296 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-04-15T15:16:30.085560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237476,"kb_used_data":4660,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737244,"statfs":{"total":773094113280,"available":772850937856,"internally_reserved":0,"allocated":4771840,"data_stored":2694064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000792"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884133+0000","last_change":"2024-04-15T15:16:25.883054+0000","last_active":"2024-04-15T15:16:26.884133+0000","last_peered":"2024-04-15T15:16:26.884133+0000","last_clean":"2024-04-15T15:16:26.884133+0000","last_became_active":"2024-04-15T15:16:25.882744+0000","last_became_peered":"2024-04-15T15:16:25.882744+0000","last_unstale":"2024-04-15T15:16:26.884133+0000","last_undegraded":"2024-04-15T15:16:26.884133+0000","last_fullsized":"2024-04-15T15:16:26.884133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:06:55.204459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883810+0000","last_change":"2024-04-15T15:16:25.888729+0000","last_active":"2024-04-15T15:16:26.883810+0000","last_peered":"2024-04-15T15:16:26.883810+0000","last_clean":"2024-04-15T15:16:26.883810+0000","last_became_active":"2024-04-15T15:16:25.888450+0000","last_became_peered":"2024-04-15T15:16:25.888450+0000","last_unstale":"2024-04-15T15:16:26.883810+0000","last_undegraded":"2024-04-15T15:16:26.883810+0000","last_fullsized":"2024-04-15T15:16:26.883810+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902977+0000","last_change":"2024-04-15T15:16:25.902269+0000","last_active":"2024-04-15T15:16:25.902977+0000","last_peered":"2024-04-15T15:16:25.902977+0000","last_clean":"2024-04-15T15:16:25.902977+0000","last_became_active":"2024-04-15T15:16:25.901972+0000","last_became_peered":"2024-04-15T15:16:25.901972+0000","last_unstale":"2024-04-15T15:16:25.902977+0000","last_undegraded":"2024-04-15T15:16:25.902977+0000","last_fullsized":"2024-04-15T15:16:25.902977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898887+0000","last_change":"2024-04-15T15:16:25.897033+0000","last_active":"2024-04-15T15:16:25.898887+0000","last_peered":"2024-04-15T15:16:25.898887+0000","last_clean":"2024-04-15T15:16:25.898887+0000","last_became_active":"2024-04-15T15:16:25.896432+0000","last_became_peered":"2024-04-15T15:16:25.896432+0000","last_unstale":"2024-04-15T15:16:25.898887+0000","last_undegraded":"2024-04-15T15:16:25.898887+0000","last_fullsized":"2024-04-15T15:16:25.898887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:19.565523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891565+0000","last_change":"2024-04-15T15:16:25.889069+0000","last_active":"2024-04-15T15:16:25.891565+0000","last_peered":"2024-04-15T15:16:25.891565+0000","last_clean":"2024-04-15T15:16:25.891565+0000","last_became_active":"2024-04-15T15:16:25.888793+0000","last_became_peered":"2024-04-15T15:16:25.888793+0000","last_unstale":"2024-04-15T15:16:25.891565+0000","last_undegraded":"2024-04-15T15:16:25.891565+0000","last_fullsized":"2024-04-15T15:16:25.891565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:48.611396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888862+0000","last_change":"2024-04-15T15:16:25.887072+0000","last_active":"2024-04-15T15:16:25.888862+0000","last_peered":"2024-04-15T15:16:25.888862+0000","last_clean":"2024-04-15T15:16:25.888862+0000","last_became_active":"2024-04-15T15:16:25.886821+0000","last_became_peered":"2024-04-15T15:16:25.886821+0000","last_unstale":"2024-04-15T15:16:25.888862+0000","last_undegraded":"2024-04-15T15:16:25.888862+0000","last_fullsized":"2024-04-15T15:16:25.888862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904107+0000","last_change":"2024-04-15T15:16:25.900378+0000","last_active":"2024-04-15T15:16:25.904107+0000","last_peered":"2024-04-15T15:16:25.904107+0000","last_clean":"2024-04-15T15:16:25.904107+0000","last_became_active":"2024-04-15T15:16:25.900071+0000","last_became_peered":"2024-04-15T15:16:25.900071+0000","last_unstale":"2024-04-15T15:16:25.904107+0000","last_undegraded":"2024-04-15T15:16:25.904107+0000","last_fullsized":"2024-04-15T15:16:25.904107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:43.288974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883618+0000","last_change":"2024-04-15T15:16:25.893507+0000","last_active":"2024-04-15T15:16:26.883618+0000","last_peered":"2024-04-15T15:16:26.883618+0000","last_clean":"2024-04-15T15:16:26.883618+0000","last_became_active":"2024-04-15T15:16:25.893005+0000","last_became_peered":"2024-04-15T15:16:25.893005+0000","last_unstale":"2024-04-15T15:16:26.883618+0000","last_undegraded":"2024-04-15T15:16:26.883618+0000","last_fullsized":"2024-04-15T15:16:26.883618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.895340+0000","last_change":"2024-04-15T15:16:25.894870+0000","last_active":"2024-04-15T15:16:25.895340+0000","last_peered":"2024-04-15T15:16:25.895340+0000","last_clean":"2024-04-15T15:16:25.895340+0000","last_became_active":"2024-04-15T15:16:25.894343+0000","last_became_peered":"2024-04-15T15:16:25.894343+0000","last_unstale":"2024-04-15T15:16:25.895340+0000","last_undegraded":"2024-04-15T15:16:25.895340+0000","last_fullsized":"2024-04-15T15:16:25.895340+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904248+0000","last_change":"2024-04-15T15:16:25.900279+0000","last_active":"2024-04-15T15:16:25.904248+0000","last_peered":"2024-04-15T15:16:25.904248+0000","last_clean":"2024-04-15T15:16:25.904248+0000","last_became_active":"2024-04-15T15:16:25.899969+0000","last_became_peered":"2024-04-15T15:16:25.899969+0000","last_unstale":"2024-04-15T15:16:25.904248+0000","last_undegraded":"2024-04-15T15:16:25.904248+0000","last_fullsized":"2024-04-15T15:16:25.904248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:19:11.196336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911723+0000","last_change":"2024-04-15T15:16:25.904761+0000","last_active":"2024-04-15T15:16:25.911723+0000","last_peered":"2024-04-15T15:16:25.911723+0000","last_clean":"2024-04-15T15:16:25.911723+0000","last_became_active":"2024-04-15T15:16:25.904477+0000","last_became_peered":"2024-04-15T15:16:25.904477+0000","last_unstale":"2024-04-15T15:16:25.911723+0000","last_undegraded":"2024-04-15T15:16:25.911723+0000","last_fullsized":"2024-04-15T15:16:25.911723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884502+0000","last_change":"2024-04-15T15:16:25.882646+0000","last_active":"2024-04-15T15:16:25.884502+0000","last_peered":"2024-04-15T15:16:25.884502+0000","last_clean":"2024-04-15T15:16:25.884502+0000","last_became_active":"2024-04-15T15:16:25.882375+0000","last_became_peered":"2024-04-15T15:16:25.882375+0000","last_unstale":"2024-04-15T15:16:25.884502+0000","last_undegraded":"2024-04-15T15:16:25.884502+0000","last_fullsized":"2024-04-15T15:16:25.884502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906634+0000","last_change":"2024-04-15T15:16:25.901003+0000","last_active":"2024-04-15T15:16:25.906634+0000","last_peered":"2024-04-15T15:16:25.906634+0000","last_clean":"2024-04-15T15:16:25.906634+0000","last_became_active":"2024-04-15T15:16:25.900729+0000","last_became_peered":"2024-04-15T15:16:25.900729+0000","last_unstale":"2024-04-15T15:16:25.906634+0000","last_undegraded":"2024-04-15T15:16:25.906634+0000","last_fullsized":"2024-04-15T15:16:25.906634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:34.769819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885365+0000","last_change":"2024-04-15T15:16:25.899260+0000","last_active":"2024-04-15T15:16:26.885365+0000","last_peered":"2024-04-15T15:16:26.885365+0000","last_clean":"2024-04-15T15:16:26.885365+0000","last_became_active":"2024-04-15T15:16:25.898979+0000","last_became_peered":"2024-04-15T15:16:25.898979+0000","last_unstale":"2024-04-15T15:16:26.885365+0000","last_undegraded":"2024-04-15T15:16:26.885365+0000","last_fullsized":"2024-04-15T15:16:26.885365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904064+0000","last_change":"2024-04-15T15:16:25.899744+0000","last_active":"2024-04-15T15:16:25.904064+0000","last_peered":"2024-04-15T15:16:25.904064+0000","last_clean":"2024-04-15T15:16:25.904064+0000","last_became_active":"2024-04-15T15:16:25.899472+0000","last_became_peered":"2024-04-15T15:16:25.899472+0000","last_unstale":"2024-04-15T15:16:25.904064+0000","last_undegraded":"2024-04-15T15:16:25.904064+0000","last_fullsized":"2024-04-15T15:16:25.904064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185213+0000","last_change":"2024-04-15T15:16:25.887925+0000","last_active":"2024-04-15T15:16:27.185213+0000","last_peered":"2024-04-15T15:16:27.185213+0000","last_clean":"2024-04-15T15:16:27.185213+0000","last_became_active":"2024-04-15T15:16:25.887628+0000","last_became_peered":"2024-04-15T15:16:25.887628+0000","last_unstale":"2024-04-15T15:16:27.185213+0000","last_undegraded":"2024-04-15T15:16:27.185213+0000","last_fullsized":"2024-04-15T15:16:27.185213+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185828+0000","last_change":"2024-04-15T15:16:25.902349+0000","last_active":"2024-04-15T15:16:27.185828+0000","last_peered":"2024-04-15T15:16:27.185828+0000","last_clean":"2024-04-15T15:16:27.185828+0000","last_became_active":"2024-04-15T15:16:25.901988+0000","last_became_peered":"2024-04-15T15:16:25.901988+0000","last_unstale":"2024-04-15T15:16:27.185828+0000","last_undegraded":"2024-04-15T15:16:27.185828+0000","last_fullsized":"2024-04-15T15:16:27.185828+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885281+0000","last_change":"2024-04-15T15:16:25.892123+0000","last_active":"2024-04-15T15:16:26.885281+0000","last_peered":"2024-04-15T15:16:26.885281+0000","last_clean":"2024-04-15T15:16:26.885281+0000","last_became_active":"2024-04-15T15:16:25.891853+0000","last_became_peered":"2024-04-15T15:16:25.891853+0000","last_unstale":"2024-04-15T15:16:26.885281+0000","last_undegraded":"2024-04-15T15:16:26.885281+0000","last_fullsized":"2024-04-15T15:16:26.885281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884688+0000","last_change":"2024-04-15T15:16:25.880039+0000","last_active":"2024-04-15T15:16:25.884688+0000","last_peered":"2024-04-15T15:16:25.884688+0000","last_clean":"2024-04-15T15:16:25.884688+0000","last_became_active":"2024-04-15T15:16:25.879753+0000","last_became_peered":"2024-04-15T15:16:25.879753+0000","last_unstale":"2024-04-15T15:16:25.884688+0000","last_undegraded":"2024-04-15T15:16:25.884688+0000","last_fullsized":"2024-04-15T15:16:25.884688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:58:15.988048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885269+0000","last_change":"2024-04-15T15:16:25.897291+0000","last_active":"2024-04-15T15:16:26.885269+0000","last_peered":"2024-04-15T15:16:26.885269+0000","last_clean":"2024-04-15T15:16:26.885269+0000","last_became_active":"2024-04-15T15:16:25.896977+0000","last_became_peered":"2024-04-15T15:16:25.896977+0000","last_unstale":"2024-04-15T15:16:26.885269+0000","last_undegraded":"2024-04-15T15:16:26.885269+0000","last_fullsized":"2024-04-15T15:16:26.885269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901153+0000","last_change":"2024-04-15T15:16:25.889177+0000","last_active":"2024-04-15T15:16:25.901153+0000","last_peered":"2024-04-15T15:16:25.901153+0000","last_clean":"2024-04-15T15:16:25.901153+0000","last_became_active":"2024-04-15T15:16:25.888843+0000","last_became_peered":"2024-04-15T15:16:25.888843+0000","last_unstale":"2024-04-15T15:16:25.901153+0000","last_undegraded":"2024-04-15T15:16:25.901153+0000","last_fullsized":"2024-04-15T15:16:25.901153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885679+0000","last_change":"2024-04-15T15:16:25.892453+0000","last_active":"2024-04-15T15:16:26.885679+0000","last_peered":"2024-04-15T15:16:26.885679+0000","last_clean":"2024-04-15T15:16:26.885679+0000","last_became_active":"2024-04-15T15:16:25.892168+0000","last_became_peered":"2024-04-15T15:16:25.892168+0000","last_unstale":"2024-04-15T15:16:26.885679+0000","last_undegraded":"2024-04-15T15:16:26.885679+0000","last_fullsized":"2024-04-15T15:16:26.885679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185288+0000","last_change":"2024-04-15T15:16:25.893780+0000","last_active":"2024-04-15T15:16:27.185288+0000","last_peered":"2024-04-15T15:16:27.185288+0000","last_clean":"2024-04-15T15:16:27.185288+0000","last_became_active":"2024-04-15T15:16:25.893529+0000","last_became_peered":"2024-04-15T15:16:25.893529+0000","last_unstale":"2024-04-15T15:16:27.185288+0000","last_undegraded":"2024-04-15T15:16:27.185288+0000","last_fullsized":"2024-04-15T15:16:27.185288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891390+0000","last_change":"2024-04-15T15:16:25.885032+0000","last_active":"2024-04-15T15:16:25.891390+0000","last_peered":"2024-04-15T15:16:25.891390+0000","last_clean":"2024-04-15T15:16:25.891390+0000","last_became_active":"2024-04-15T15:16:25.884397+0000","last_became_peered":"2024-04-15T15:16:25.884397+0000","last_unstale":"2024-04-15T15:16:25.891390+0000","last_undegraded":"2024-04-15T15:16:25.891390+0000","last_fullsized":"2024-04-15T15:16:25.891390+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:31.155865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885450+0000","last_change":"2024-04-15T15:16:25.882933+0000","last_active":"2024-04-15T15:16:26.885450+0000","last_peered":"2024-04-15T15:16:26.885450+0000","last_clean":"2024-04-15T15:16:26.885450+0000","last_became_active":"2024-04-15T15:16:25.882643+0000","last_became_peered":"2024-04-15T15:16:25.882643+0000","last_unstale":"2024-04-15T15:16:26.885450+0000","last_undegraded":"2024-04-15T15:16:26.885450+0000","last_fullsized":"2024-04-15T15:16:26.885450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891210+0000","last_change":"2024-04-15T15:16:25.890339+0000","last_active":"2024-04-15T15:16:25.891210+0000","last_peered":"2024-04-15T15:16:25.891210+0000","last_clean":"2024-04-15T15:16:25.891210+0000","last_became_active":"2024-04-15T15:16:25.889922+0000","last_became_peered":"2024-04-15T15:16:25.889922+0000","last_unstale":"2024-04-15T15:16:25.891210+0000","last_undegraded":"2024-04-15T15:16:25.891210+0000","last_fullsized":"2024-04-15T15:16:25.891210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885230+0000","last_change":"2024-04-15T15:16:25.890968+0000","last_active":"2024-04-15T15:16:26.885230+0000","last_peered":"2024-04-15T15:16:26.885230+0000","last_clean":"2024-04-15T15:16:26.885230+0000","last_became_active":"2024-04-15T15:16:25.890686+0000","last_became_peered":"2024-04-15T15:16:25.890686+0000","last_unstale":"2024-04-15T15:16:26.885230+0000","last_undegraded":"2024-04-15T15:16:26.885230+0000","last_fullsized":"2024-04-15T15:16:26.885230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904156+0000","last_change":"2024-04-15T15:16:25.900260+0000","last_active":"2024-04-15T15:16:25.904156+0000","last_peered":"2024-04-15T15:16:25.904156+0000","last_clean":"2024-04-15T15:16:25.904156+0000","last_became_active":"2024-04-15T15:16:25.899948+0000","last_became_peered":"2024-04-15T15:16:25.899948+0000","last_unstale":"2024-04-15T15:16:25.904156+0000","last_undegraded":"2024-04-15T15:16:25.904156+0000","last_fullsized":"2024-04-15T15:16:25.904156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884989+0000","last_change":"2024-04-15T15:16:25.884088+0000","last_active":"2024-04-15T15:16:26.884989+0000","last_peered":"2024-04-15T15:16:26.884989+0000","last_clean":"2024-04-15T15:16:26.884989+0000","last_became_active":"2024-04-15T15:16:25.883804+0000","last_became_peered":"2024-04-15T15:16:25.883804+0000","last_unstale":"2024-04-15T15:16:26.884989+0000","last_undegraded":"2024-04-15T15:16:26.884989+0000","last_fullsized":"2024-04-15T15:16:26.884989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885622+0000","last_change":"2024-04-15T15:16:25.879663+0000","last_active":"2024-04-15T15:16:26.885622+0000","last_peered":"2024-04-15T15:16:26.885622+0000","last_clean":"2024-04-15T15:16:26.885622+0000","last_became_active":"2024-04-15T15:16:25.879385+0000","last_became_peered":"2024-04-15T15:16:25.879385+0000","last_unstale":"2024-04-15T15:16:26.885622+0000","last_undegraded":"2024-04-15T15:16:26.885622+0000","last_fullsized":"2024-04-15T15:16:26.885622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185437+0000","last_change":"2024-04-15T15:16:25.893793+0000","last_active":"2024-04-15T15:16:27.185437+0000","last_peered":"2024-04-15T15:16:27.185437+0000","last_clean":"2024-04-15T15:16:27.185437+0000","last_became_active":"2024-04-15T15:16:25.893556+0000","last_became_peered":"2024-04-15T15:16:25.893556+0000","last_unstale":"2024-04-15T15:16:27.185437+0000","last_undegraded":"2024-04-15T15:16:27.185437+0000","last_fullsized":"2024-04-15T15:16:27.185437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888033+0000","last_change":"2024-04-15T15:16:25.886972+0000","last_active":"2024-04-15T15:16:25.888033+0000","last_peered":"2024-04-15T15:16:25.888033+0000","last_clean":"2024-04-15T15:16:25.888033+0000","last_became_active":"2024-04-15T15:16:25.886718+0000","last_became_peered":"2024-04-15T15:16:25.886718+0000","last_unstale":"2024-04-15T15:16:25.888033+0000","last_undegraded":"2024-04-15T15:16:25.888033+0000","last_fullsized":"2024-04-15T15:16:25.888033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185663+0000","last_change":"2024-04-15T15:16:25.896046+0000","last_active":"2024-04-15T15:16:27.185663+0000","last_peered":"2024-04-15T15:16:27.185663+0000","last_clean":"2024-04-15T15:16:27.185663+0000","last_became_active":"2024-04-15T15:16:25.895768+0000","last_became_peered":"2024-04-15T15:16:25.895768+0000","last_unstale":"2024-04-15T15:16:27.185663+0000","last_undegraded":"2024-04-15T15:16:27.185663+0000","last_fullsized":"2024-04-15T15:16:27.185663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885577+0000","last_change":"2024-04-15T15:16:24.886178+0000","last_active":"2024-04-15T15:16:26.885577+0000","last_peered":"2024-04-15T15:16:26.885577+0000","last_clean":"2024-04-15T15:16:26.885577+0000","last_became_active":"2024-04-15T15:16:24.885910+0000","last_became_peered":"2024-04-15T15:16:24.885910+0000","last_unstale":"2024-04-15T15:16:26.885577+0000","last_undegraded":"2024-04-15T15:16:26.885577+0000","last_fullsized":"2024-04-15T15:16:26.885577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903528+0000","last_change":"2024-04-15T15:16:25.902820+0000","last_active":"2024-04-15T15:16:25.903528+0000","last_peered":"2024-04-15T15:16:25.903528+0000","last_clean":"2024-04-15T15:16:25.903528+0000","last_became_active":"2024-04-15T15:16:25.902552+0000","last_became_peered":"2024-04-15T15:16:25.902552+0000","last_unstale":"2024-04-15T15:16:25.903528+0000","last_undegraded":"2024-04-15T15:16:25.903528+0000","last_fullsized":"2024-04-15T15:16:25.903528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898338+0000","last_change":"2024-04-15T15:16:24.886331+0000","last_active":"2024-04-15T15:16:25.898338+0000","last_peered":"2024-04-15T15:16:25.898338+0000","last_clean":"2024-04-15T15:16:25.898338+0000","last_became_active":"2024-04-15T15:16:24.886052+0000","last_became_peered":"2024-04-15T15:16:24.886052+0000","last_unstale":"2024-04-15T15:16:25.898338+0000","last_undegraded":"2024-04-15T15:16:25.898338+0000","last_fullsized":"2024-04-15T15:16:25.898338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.899883+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.899620+0000","last_became_peered":"2024-04-15T15:16:25.899620+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898694+0000","last_change":"2024-04-15T15:16:24.890646+0000","last_active":"2024-04-15T15:16:25.898694+0000","last_peered":"2024-04-15T15:16:25.898694+0000","last_clean":"2024-04-15T15:16:25.898694+0000","last_became_active":"2024-04-15T15:16:24.890362+0000","last_became_peered":"2024-04-15T15:16:24.890362+0000","last_unstale":"2024-04-15T15:16:25.898694+0000","last_undegraded":"2024-04-15T15:16:25.898694+0000","last_fullsized":"2024-04-15T15:16:25.898694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883847+0000","last_change":"2024-04-15T15:16:25.895423+0000","last_active":"2024-04-15T15:16:26.883847+0000","last_peered":"2024-04-15T15:16:26.883847+0000","last_clean":"2024-04-15T15:16:26.883847+0000","last_became_active":"2024-04-15T15:16:25.889905+0000","last_became_peered":"2024-04-15T15:16:25.889905+0000","last_unstale":"2024-04-15T15:16:26.883847+0000","last_undegraded":"2024-04-15T15:16:26.883847+0000","last_fullsized":"2024-04-15T15:16:26.883847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:50:15.175711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886111+0000","last_change":"2024-04-15T15:16:24.887352+0000","last_active":"2024-04-15T15:16:26.886111+0000","last_peered":"2024-04-15T15:16:26.886111+0000","last_clean":"2024-04-15T15:16:26.886111+0000","last_became_active":"2024-04-15T15:16:24.887054+0000","last_became_peered":"2024-04-15T15:16:24.887054+0000","last_unstale":"2024-04-15T15:16:26.886111+0000","last_undegraded":"2024-04-15T15:16:26.886111+0000","last_fullsized":"2024-04-15T15:16:26.886111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.882206+0000","last_change":"2024-04-15T15:16:25.881247+0000","last_active":"2024-04-15T15:16:25.882206+0000","last_peered":"2024-04-15T15:16:25.882206+0000","last_clean":"2024-04-15T15:16:25.882206+0000","last_became_active":"2024-04-15T15:16:25.880981+0000","last_became_peered":"2024-04-15T15:16:25.880981+0000","last_unstale":"2024-04-15T15:16:25.882206+0000","last_undegraded":"2024-04-15T15:16:25.882206+0000","last_fullsized":"2024-04-15T15:16:25.882206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903854+0000","last_change":"2024-04-15T15:16:24.891022+0000","last_active":"2024-04-15T15:16:25.903854+0000","last_peered":"2024-04-15T15:16:25.903854+0000","last_clean":"2024-04-15T15:16:25.903854+0000","last_became_active":"2024-04-15T15:16:24.890770+0000","last_became_peered":"2024-04-15T15:16:24.890770+0000","last_unstale":"2024-04-15T15:16:25.903854+0000","last_undegraded":"2024-04-15T15:16:25.903854+0000","last_fullsized":"2024-04-15T15:16:25.903854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901334+0000","last_change":"2024-04-15T15:16:25.893573+0000","last_active":"2024-04-15T15:16:25.901334+0000","last_peered":"2024-04-15T15:16:25.901334+0000","last_clean":"2024-04-15T15:16:25.901334+0000","last_became_active":"2024-04-15T15:16:25.893321+0000","last_became_peered":"2024-04-15T15:16:25.893321+0000","last_unstale":"2024-04-15T15:16:25.901334+0000","last_undegraded":"2024-04-15T15:16:25.901334+0000","last_fullsized":"2024-04-15T15:16:25.901334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185361+0000","last_change":"2024-04-15T15:16:24.889597+0000","last_active":"2024-04-15T15:16:27.185361+0000","last_peered":"2024-04-15T15:16:27.185361+0000","last_clean":"2024-04-15T15:16:27.185361+0000","last_became_active":"2024-04-15T15:16:24.889176+0000","last_became_peered":"2024-04-15T15:16:24.889176+0000","last_unstale":"2024-04-15T15:16:27.185361+0000","last_undegraded":"2024-04-15T15:16:27.185361+0000","last_fullsized":"2024-04-15T15:16:27.185361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885832+0000","last_change":"2024-04-15T15:16:25.904897+0000","last_active":"2024-04-15T15:16:26.885832+0000","last_peered":"2024-04-15T15:16:26.885832+0000","last_clean":"2024-04-15T15:16:26.885832+0000","last_became_active":"2024-04-15T15:16:25.904610+0000","last_became_peered":"2024-04-15T15:16:25.904610+0000","last_unstale":"2024-04-15T15:16:26.885832+0000","last_undegraded":"2024-04-15T15:16:26.885832+0000","last_fullsized":"2024-04-15T15:16:26.885832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876235+0000","last_change":"2024-04-15T15:16:24.891428+0000","last_active":"2024-04-15T15:16:25.876235+0000","last_peered":"2024-04-15T15:16:25.876235+0000","last_clean":"2024-04-15T15:16:25.876235+0000","last_became_active":"2024-04-15T15:16:24.891151+0000","last_became_peered":"2024-04-15T15:16:24.891151+0000","last_unstale":"2024-04-15T15:16:25.876235+0000","last_undegraded":"2024-04-15T15:16:25.876235+0000","last_fullsized":"2024-04-15T15:16:25.876235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:56:54.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883767+0000","last_change":"2024-04-15T15:16:25.898140+0000","last_active":"2024-04-15T15:16:26.883767+0000","last_peered":"2024-04-15T15:16:26.883767+0000","last_clean":"2024-04-15T15:16:26.883767+0000","last_became_active":"2024-04-15T15:16:25.897492+0000","last_became_peered":"2024-04-15T15:16:25.897492+0000","last_unstale":"2024-04-15T15:16:26.883767+0000","last_undegraded":"2024-04-15T15:16:26.883767+0000","last_fullsized":"2024-04-15T15:16:26.883767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884775+0000","last_change":"2024-04-15T15:16:24.884571+0000","last_active":"2024-04-15T15:16:26.884775+0000","last_peered":"2024-04-15T15:16:26.884775+0000","last_clean":"2024-04-15T15:16:26.884775+0000","last_became_active":"2024-04-15T15:16:24.884274+0000","last_became_peered":"2024-04-15T15:16:24.884274+0000","last_unstale":"2024-04-15T15:16:26.884775+0000","last_undegraded":"2024-04-15T15:16:26.884775+0000","last_fullsized":"2024-04-15T15:16:26.884775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.899098+0000","last_change":"2024-04-15T15:16:25.898346+0000","last_active":"2024-04-15T15:16:25.899098+0000","last_peered":"2024-04-15T15:16:25.899098+0000","last_clean":"2024-04-15T15:16:25.899098+0000","last_became_active":"2024-04-15T15:16:25.897145+0000","last_became_peered":"2024-04-15T15:16:25.897145+0000","last_unstale":"2024-04-15T15:16:25.899098+0000","last_undegraded":"2024-04-15T15:16:25.899098+0000","last_fullsized":"2024-04-15T15:16:25.899098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:54.775491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884135+0000","last_change":"2024-04-15T15:16:24.888936+0000","last_active":"2024-04-15T15:16:26.884135+0000","last_peered":"2024-04-15T15:16:26.884135+0000","last_clean":"2024-04-15T15:16:26.884135+0000","last_became_active":"2024-04-15T15:16:24.888050+0000","last_became_peered":"2024-04-15T15:16:24.888050+0000","last_unstale":"2024-04-15T15:16:26.884135+0000","last_undegraded":"2024-04-15T15:16:26.884135+0000","last_fullsized":"2024-04-15T15:16:26.884135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185633+0000","last_change":"2024-04-15T15:16:25.899963+0000","last_active":"2024-04-15T15:16:27.185633+0000","last_peered":"2024-04-15T15:16:27.185633+0000","last_clean":"2024-04-15T15:16:27.185633+0000","last_became_active":"2024-04-15T15:16:25.899676+0000","last_became_peered":"2024-04-15T15:16:25.899676+0000","last_unstale":"2024-04-15T15:16:27.185633+0000","last_undegraded":"2024-04-15T15:16:27.185633+0000","last_fullsized":"2024-04-15T15:16:27.185633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.886799+0000","last_change":"2024-04-15T15:16:24.887199+0000","last_active":"2024-04-15T15:16:25.886799+0000","last_peered":"2024-04-15T15:16:25.886799+0000","last_clean":"2024-04-15T15:16:25.886799+0000","last_became_active":"2024-04-15T15:16:24.886945+0000","last_became_peered":"2024-04-15T15:16:24.886945+0000","last_unstale":"2024-04-15T15:16:25.886799+0000","last_undegraded":"2024-04-15T15:16:25.886799+0000","last_fullsized":"2024-04-15T15:16:25.886799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.893867+0000","last_change":"2024-04-15T15:16:25.892203+0000","last_active":"2024-04-15T15:16:25.893867+0000","last_peered":"2024-04-15T15:16:25.893867+0000","last_clean":"2024-04-15T15:16:25.893867+0000","last_became_active":"2024-04-15T15:16:25.891899+0000","last_became_peered":"2024-04-15T15:16:25.891899+0000","last_unstale":"2024-04-15T15:16:25.893867+0000","last_undegraded":"2024-04-15T15:16:25.893867+0000","last_fullsized":"2024-04-15T15:16:25.893867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904566+0000","last_change":"2024-04-15T15:16:24.890265+0000","last_active":"2024-04-15T15:16:25.904566+0000","last_peered":"2024-04-15T15:16:25.904566+0000","last_clean":"2024-04-15T15:16:25.904566+0000","last_became_active":"2024-04-15T15:16:24.889963+0000","last_became_peered":"2024-04-15T15:16:24.889963+0000","last_unstale":"2024-04-15T15:16:25.904566+0000","last_undegraded":"2024-04-15T15:16:25.904566+0000","last_fullsized":"2024-04-15T15:16:25.904566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886503+0000","last_change":"2024-04-15T15:16:25.893921+0000","last_active":"2024-04-15T15:16:26.886503+0000","last_peered":"2024-04-15T15:16:26.886503+0000","last_clean":"2024-04-15T15:16:26.886503+0000","last_became_active":"2024-04-15T15:16:25.892154+0000","last_became_peered":"2024-04-15T15:16:25.892154+0000","last_unstale":"2024-04-15T15:16:26.886503+0000","last_undegraded":"2024-04-15T15:16:26.886503+0000","last_fullsized":"2024-04-15T15:16:26.886503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884185+0000","last_change":"2024-04-15T15:16:24.889313+0000","last_active":"2024-04-15T15:16:26.884185+0000","last_peered":"2024-04-15T15:16:26.884185+0000","last_clean":"2024-04-15T15:16:26.884185+0000","last_became_active":"2024-04-15T15:16:24.889037+0000","last_became_peered":"2024-04-15T15:16:24.889037+0000","last_unstale":"2024-04-15T15:16:26.884185+0000","last_undegraded":"2024-04-15T15:16:26.884185+0000","last_fullsized":"2024-04-15T15:16:26.884185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:58.773260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.889667+0000","last_change":"2024-04-15T15:16:25.888305+0000","last_active":"2024-04-15T15:16:25.889667+0000","last_peered":"2024-04-15T15:16:25.889667+0000","last_clean":"2024-04-15T15:16:25.889667+0000","last_became_active":"2024-04-15T15:16:25.888053+0000","last_became_peered":"2024-04-15T15:16:25.888053+0000","last_unstale":"2024-04-15T15:16:25.889667+0000","last_undegraded":"2024-04-15T15:16:25.889667+0000","last_fullsized":"2024-04-15T15:16:25.889667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884136+0000","last_change":"2024-04-15T15:16:24.882824+0000","last_active":"2024-04-15T15:16:26.884136+0000","last_peered":"2024-04-15T15:16:26.884136+0000","last_clean":"2024-04-15T15:16:26.884136+0000","last_became_active":"2024-04-15T15:16:24.882560+0000","last_became_peered":"2024-04-15T15:16:24.882560+0000","last_unstale":"2024-04-15T15:16:26.884136+0000","last_undegraded":"2024-04-15T15:16:26.884136+0000","last_fullsized":"2024-04-15T15:16:26.884136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:08.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883781+0000","last_change":"2024-04-15T15:16:25.883512+0000","last_active":"2024-04-15T15:16:26.883781+0000","last_peered":"2024-04-15T15:16:26.883781+0000","last_clean":"2024-04-15T15:16:26.883781+0000","last_became_active":"2024-04-15T15:16:25.883253+0000","last_became_peered":"2024-04-15T15:16:25.883253+0000","last_unstale":"2024-04-15T15:16:26.883781+0000","last_undegraded":"2024-04-15T15:16:26.883781+0000","last_fullsized":"2024-04-15T15:16:26.883781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.877572+0000","last_change":"2024-04-15T15:16:24.891194+0000","last_active":"2024-04-15T15:16:25.877572+0000","last_peered":"2024-04-15T15:16:25.877572+0000","last_clean":"2024-04-15T15:16:25.877572+0000","last_became_active":"2024-04-15T15:16:24.890944+0000","last_became_peered":"2024-04-15T15:16:24.890944+0000","last_unstale":"2024-04-15T15:16:25.877572+0000","last_undegraded":"2024-04-15T15:16:25.877572+0000","last_fullsized":"2024-04-15T15:16:25.877572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185589+0000","last_change":"2024-04-15T15:16:25.893190+0000","last_active":"2024-04-15T15:16:27.185589+0000","last_peered":"2024-04-15T15:16:27.185589+0000","last_clean":"2024-04-15T15:16:27.185589+0000","last_became_active":"2024-04-15T15:16:25.892917+0000","last_became_peered":"2024-04-15T15:16:25.892917+0000","last_unstale":"2024-04-15T15:16:27.185589+0000","last_undegraded":"2024-04-15T15:16:27.185589+0000","last_fullsized":"2024-04-15T15:16:27.185589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:43:41.370786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885147+0000","last_change":"2024-04-15T15:16:24.886029+0000","last_active":"2024-04-15T15:16:26.885147+0000","last_peered":"2024-04-15T15:16:26.885147+0000","last_clean":"2024-04-15T15:16:26.885147+0000","last_became_active":"2024-04-15T15:16:24.885730+0000","last_became_peered":"2024-04-15T15:16:24.885730+0000","last_unstale":"2024-04-15T15:16:26.885147+0000","last_undegraded":"2024-04-15T15:16:26.885147+0000","last_fullsized":"2024-04-15T15:16:26.885147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884020+0000","last_change":"2024-04-15T15:16:25.882916+0000","last_active":"2024-04-15T15:16:26.884020+0000","last_peered":"2024-04-15T15:16:26.884020+0000","last_clean":"2024-04-15T15:16:26.884020+0000","last_became_active":"2024-04-15T15:16:25.882630+0000","last_became_peered":"2024-04-15T15:16:25.882630+0000","last_unstale":"2024-04-15T15:16:26.884020+0000","last_undegraded":"2024-04-15T15:16:26.884020+0000","last_fullsized":"2024-04-15T15:16:26.884020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:24.310942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185272+0000","last_change":"2024-04-15T15:16:24.889751+0000","last_active":"2024-04-15T15:16:27.185272+0000","last_peered":"2024-04-15T15:16:27.185272+0000","last_clean":"2024-04-15T15:16:27.185272+0000","last_became_active":"2024-04-15T15:16:24.889428+0000","last_became_peered":"2024-04-15T15:16:24.889428+0000","last_unstale":"2024-04-15T15:16:27.185272+0000","last_undegraded":"2024-04-15T15:16:27.185272+0000","last_fullsized":"2024-04-15T15:16:27.185272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884117+0000","last_change":"2024-04-15T15:16:25.890929+0000","last_active":"2024-04-15T15:16:26.884117+0000","last_peered":"2024-04-15T15:16:26.884117+0000","last_clean":"2024-04-15T15:16:26.884117+0000","last_became_active":"2024-04-15T15:16:25.890256+0000","last_became_peered":"2024-04-15T15:16:25.890256+0000","last_unstale":"2024-04-15T15:16:26.884117+0000","last_undegraded":"2024-04-15T15:16:26.884117+0000","last_fullsized":"2024-04-15T15:16:26.884117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:11.834293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906758+0000","last_change":"2024-04-15T15:16:24.893507+0000","last_active":"2024-04-15T15:16:25.906758+0000","last_peered":"2024-04-15T15:16:25.906758+0000","last_clean":"2024-04-15T15:16:25.906758+0000","last_became_active":"2024-04-15T15:16:24.893262+0000","last_became_peered":"2024-04-15T15:16:24.893262+0000","last_unstale":"2024-04-15T15:16:25.906758+0000","last_undegraded":"2024-04-15T15:16:25.906758+0000","last_fullsized":"2024-04-15T15:16:25.906758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:15.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911705+0000","last_change":"2024-04-15T15:16:25.904384+0000","last_active":"2024-04-15T15:16:25.911705+0000","last_peered":"2024-04-15T15:16:25.911705+0000","last_clean":"2024-04-15T15:16:25.911705+0000","last_became_active":"2024-04-15T15:16:25.904095+0000","last_became_peered":"2024-04-15T15:16:25.904095+0000","last_unstale":"2024-04-15T15:16:25.911705+0000","last_undegraded":"2024-04-15T15:16:25.911705+0000","last_fullsized":"2024-04-15T15:16:25.911705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886583+0000","last_change":"2024-04-15T15:16:24.891423+0000","last_active":"2024-04-15T15:16:26.886583+0000","last_peered":"2024-04-15T15:16:26.886583+0000","last_clean":"2024-04-15T15:16:26.886583+0000","last_became_active":"2024-04-15T15:16:24.891048+0000","last_became_peered":"2024-04-15T15:16:24.891048+0000","last_unstale":"2024-04-15T15:16:26.886583+0000","last_undegraded":"2024-04-15T15:16:26.886583+0000","last_fullsized":"2024-04-15T15:16:26.886583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:47:10.367180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.894040+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.893781+0000","last_became_peered":"2024-04-15T15:16:25.893781+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.897493+0000","last_change":"2024-04-15T15:16:24.889598+0000","last_active":"2024-04-15T15:16:25.897493+0000","last_peered":"2024-04-15T15:16:25.897493+0000","last_clean":"2024-04-15T15:16:25.897493+0000","last_became_active":"2024-04-15T15:16:24.889298+0000","last_became_peered":"2024-04-15T15:16:24.889298+0000","last_unstale":"2024-04-15T15:16:25.897493+0000","last_undegraded":"2024-04-15T15:16:25.897493+0000","last_fullsized":"2024-04-15T15:16:25.897493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:54:18.385006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891485+0000","last_change":"2024-04-15T15:16:25.884522+0000","last_active":"2024-04-15T15:16:25.891485+0000","last_peered":"2024-04-15T15:16:25.891485+0000","last_clean":"2024-04-15T15:16:25.891485+0000","last_became_active":"2024-04-15T15:16:25.883126+0000","last_became_peered":"2024-04-15T15:16:25.883126+0000","last_unstale":"2024-04-15T15:16:25.891485+0000","last_undegraded":"2024-04-15T15:16:25.891485+0000","last_fullsized":"2024-04-15T15:16:25.891485+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:03.969456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876163+0000","last_change":"2024-04-15T15:16:24.890031+0000","last_active":"2024-04-15T15:16:25.876163+0000","last_peered":"2024-04-15T15:16:25.876163+0000","last_clean":"2024-04-15T15:16:25.876163+0000","last_became_active":"2024-04-15T15:16:24.889203+0000","last_became_peered":"2024-04-15T15:16:24.889203+0000","last_unstale":"2024-04-15T15:16:25.876163+0000","last_undegraded":"2024-04-15T15:16:25.876163+0000","last_fullsized":"2024-04-15T15:16:25.876163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:18.919033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185440+0000","last_change":"2024-04-15T15:16:25.894269+0000","last_active":"2024-04-15T15:16:27.185440+0000","last_peered":"2024-04-15T15:16:27.185440+0000","last_clean":"2024-04-15T15:16:27.185440+0000","last_became_active":"2024-04-15T15:16:25.893279+0000","last_became_peered":"2024-04-15T15:16:25.893279+0000","last_unstale":"2024-04-15T15:16:27.185440+0000","last_undegraded":"2024-04-15T15:16:27.185440+0000","last_fullsized":"2024-04-15T15:16:27.185440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:48:22.937069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891401+0000","last_change":"2024-04-15T15:16:24.877420+0000","last_active":"2024-04-15T15:16:25.891401+0000","last_peered":"2024-04-15T15:16:25.891401+0000","last_clean":"2024-04-15T15:16:25.891401+0000","last_became_active":"2024-04-15T15:16:24.877149+0000","last_became_peered":"2024-04-15T15:16:24.877149+0000","last_unstale":"2024-04-15T15:16:25.891401+0000","last_undegraded":"2024-04-15T15:16:25.891401+0000","last_fullsized":"2024-04-15T15:16:25.891401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:48.237807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.890064+0000","last_change":"2024-04-15T15:16:25.887625+0000","last_active":"2024-04-15T15:16:25.890064+0000","last_peered":"2024-04-15T15:16:25.890064+0000","last_clean":"2024-04-15T15:16:25.890064+0000","last_became_active":"2024-04-15T15:16:25.887362+0000","last_became_peered":"2024-04-15T15:16:25.887362+0000","last_unstale":"2024-04-15T15:16:25.890064+0000","last_undegraded":"2024-04-15T15:16:25.890064+0000","last_fullsized":"2024-04-15T15:16:25.890064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872610+0000","last_change":"2024-04-15T15:16:24.884150+0000","last_active":"2024-04-15T15:16:25.872610+0000","last_peered":"2024-04-15T15:16:25.872610+0000","last_clean":"2024-04-15T15:16:25.872610+0000","last_became_active":"2024-04-15T15:16:24.883908+0000","last_became_peered":"2024-04-15T15:16:24.883908+0000","last_unstale":"2024-04-15T15:16:25.872610+0000","last_undegraded":"2024-04-15T15:16:25.872610+0000","last_fullsized":"2024-04-15T15:16:25.872610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:58:35.147306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903908+0000","last_change":"2024-04-15T15:16:25.897701+0000","last_active":"2024-04-15T15:16:25.903908+0000","last_peered":"2024-04-15T15:16:25.903908+0000","last_clean":"2024-04-15T15:16:25.903908+0000","last_became_active":"2024-04-15T15:16:25.896891+0000","last_became_peered":"2024-04-15T15:16:25.896891+0000","last_unstale":"2024-04-15T15:16:25.903908+0000","last_undegraded":"2024-04-15T15:16:25.903908+0000","last_fullsized":"2024-04-15T15:16:25.903908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185493+0000","last_change":"2024-04-15T15:16:24.887471+0000","last_active":"2024-04-15T15:16:27.185493+0000","last_peered":"2024-04-15T15:16:27.185493+0000","last_clean":"2024-04-15T15:16:27.185493+0000","last_became_active":"2024-04-15T15:16:24.887165+0000","last_became_peered":"2024-04-15T15:16:24.887165+0000","last_unstale":"2024-04-15T15:16:27.185493+0000","last_undegraded":"2024-04-15T15:16:27.185493+0000","last_fullsized":"2024-04-15T15:16:27.185493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884030+0000","last_change":"2024-04-15T15:16:25.899465+0000","last_active":"2024-04-15T15:16:26.884030+0000","last_peered":"2024-04-15T15:16:26.884030+0000","last_clean":"2024-04-15T15:16:26.884030+0000","last_became_active":"2024-04-15T15:16:25.899205+0000","last_became_peered":"2024-04-15T15:16:25.899205+0000","last_unstale":"2024-04-15T15:16:26.884030+0000","last_undegraded":"2024-04-15T15:16:26.884030+0000","last_fullsized":"2024-04-15T15:16:26.884030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:42:16.777997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872575+0000","last_change":"2024-04-15T15:16:24.882714+0000","last_active":"2024-04-15T15:16:25.872575+0000","last_peered":"2024-04-15T15:16:25.872575+0000","last_clean":"2024-04-15T15:16:25.872575+0000","last_became_active":"2024-04-15T15:16:24.882460+0000","last_became_peered":"2024-04-15T15:16:24.882460+0000","last_unstale":"2024-04-15T15:16:25.872575+0000","last_undegraded":"2024-04-15T15:16:25.872575+0000","last_fullsized":"2024-04-15T15:16:25.872575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:11.251738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898832+0000","last_change":"2024-04-15T15:16:25.897148+0000","last_active":"2024-04-15T15:16:25.898832+0000","last_peered":"2024-04-15T15:16:25.898832+0000","last_clean":"2024-04-15T15:16:25.898832+0000","last_became_active":"2024-04-15T15:16:25.896866+0000","last_became_peered":"2024-04-15T15:16:25.896866+0000","last_unstale":"2024-04-15T15:16:25.898832+0000","last_undegraded":"2024-04-15T15:16:25.898832+0000","last_fullsized":"2024-04-15T15:16:25.898832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902029+0000","last_change":"2024-04-15T15:16:24.892956+0000","last_active":"2024-04-15T15:16:25.902029+0000","last_peered":"2024-04-15T15:16:25.902029+0000","last_clean":"2024-04-15T15:16:25.902029+0000","last_became_active":"2024-04-15T15:16:24.892664+0000","last_became_peered":"2024-04-15T15:16:24.892664+0000","last_unstale":"2024-04-15T15:16:25.902029+0000","last_undegraded":"2024-04-15T15:16:25.902029+0000","last_fullsized":"2024-04-15T15:16:25.902029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:56.013415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185690+0000","last_change":"2024-04-15T15:16:25.899975+0000","last_active":"2024-04-15T15:16:27.185690+0000","last_peered":"2024-04-15T15:16:27.185690+0000","last_clean":"2024-04-15T15:16:27.185690+0000","last_became_active":"2024-04-15T15:16:25.899691+0000","last_became_peered":"2024-04-15T15:16:25.899691+0000","last_unstale":"2024-04-15T15:16:27.185690+0000","last_undegraded":"2024-04-15T15:16:27.185690+0000","last_fullsized":"2024-04-15T15:16:27.185690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.873000+0000","last_change":"2024-04-15T15:16:24.890594+0000","last_active":"2024-04-15T15:16:25.873000+0000","last_peered":"2024-04-15T15:16:25.873000+0000","last_clean":"2024-04-15T15:16:25.873000+0000","last_became_active":"2024-04-15T15:16:24.890305+0000","last_became_peered":"2024-04-15T15:16:24.890305+0000","last_unstale":"2024-04-15T15:16:25.873000+0000","last_undegraded":"2024-04-15T15:16:25.873000+0000","last_fullsized":"2024-04-15T15:16:25.873000+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:34.555529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884060+0000","last_change":"2024-04-15T15:16:25.893319+0000","last_active":"2024-04-15T15:16:26.884060+0000","last_peered":"2024-04-15T15:16:26.884060+0000","last_clean":"2024-04-15T15:16:26.884060+0000","last_became_active":"2024-04-15T15:16:25.892794+0000","last_became_peered":"2024-04-15T15:16:25.892794+0000","last_unstale":"2024-04-15T15:16:26.884060+0000","last_undegraded":"2024-04-15T15:16:26.884060+0000","last_fullsized":"2024-04-15T15:16:26.884060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:42.664115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876776+0000","last_change":"2024-04-15T15:16:24.884802+0000","last_active":"2024-04-15T15:16:25.876776+0000","last_peered":"2024-04-15T15:16:25.876776+0000","last_clean":"2024-04-15T15:16:25.876776+0000","last_became_active":"2024-04-15T15:16:24.884519+0000","last_became_peered":"2024-04-15T15:16:24.884519+0000","last_unstale":"2024-04-15T15:16:25.876776+0000","last_undegraded":"2024-04-15T15:16:25.876776+0000","last_fullsized":"2024-04-15T15:16:25.876776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888791+0000","last_change":"2024-04-15T15:16:25.883444+0000","last_active":"2024-04-15T15:16:25.888791+0000","last_peered":"2024-04-15T15:16:25.888791+0000","last_clean":"2024-04-15T15:16:25.888791+0000","last_became_active":"2024-04-15T15:16:25.883191+0000","last_became_peered":"2024-04-15T15:16:25.883191+0000","last_unstale":"2024-04-15T15:16:25.888791+0000","last_undegraded":"2024-04-15T15:16:25.888791+0000","last_fullsized":"2024-04-15T15:16:25.888791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.875590+0000","last_change":"2024-04-15T15:16:24.886148+0000","last_active":"2024-04-15T15:16:25.875590+0000","last_peered":"2024-04-15T15:16:25.875590+0000","last_clean":"2024-04-15T15:16:25.875590+0000","last_became_active":"2024-04-15T15:16:24.885865+0000","last_became_peered":"2024-04-15T15:16:24.885865+0000","last_unstale":"2024-04-15T15:16:25.875590+0000","last_undegraded":"2024-04-15T15:16:25.875590+0000","last_fullsized":"2024-04-15T15:16:25.875590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888002+0000","last_change":"2024-04-15T15:16:25.886178+0000","last_active":"2024-04-15T15:16:25.888002+0000","last_peered":"2024-04-15T15:16:25.888002+0000","last_clean":"2024-04-15T15:16:25.888002+0000","last_became_active":"2024-04-15T15:16:25.885915+0000","last_became_peered":"2024-04-15T15:16:25.885915+0000","last_unstale":"2024-04-15T15:16:25.888002+0000","last_undegraded":"2024-04-15T15:16:25.888002+0000","last_fullsized":"2024-04-15T15:16:25.888002+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:33.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885719+0000","last_change":"2024-04-15T15:16:24.883958+0000","last_active":"2024-04-15T15:16:26.885719+0000","last_peered":"2024-04-15T15:16:26.885719+0000","last_clean":"2024-04-15T15:16:26.885719+0000","last_became_active":"2024-04-15T15:16:24.883528+0000","last_became_peered":"2024-04-15T15:16:24.883528+0000","last_unstale":"2024-04-15T15:16:26.885719+0000","last_undegraded":"2024-04-15T15:16:26.885719+0000","last_fullsized":"2024-04-15T15:16:26.885719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885492+0000","last_change":"2024-04-15T15:16:25.889861+0000","last_active":"2024-04-15T15:16:26.885492+0000","last_peered":"2024-04-15T15:16:26.885492+0000","last_clean":"2024-04-15T15:16:26.885492+0000","last_became_active":"2024-04-15T15:16:25.889591+0000","last_became_peered":"2024-04-15T15:16:25.889591+0000","last_unstale":"2024-04-15T15:16:26.885492+0000","last_undegraded":"2024-04-15T15:16:26.885492+0000","last_fullsized":"2024-04-15T15:16:26.885492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884867+0000","last_change":"2024-04-15T15:16:24.891429+0000","last_active":"2024-04-15T15:16:26.884867+0000","last_peered":"2024-04-15T15:16:26.884867+0000","last_clean":"2024-04-15T15:16:26.884867+0000","last_became_active":"2024-04-15T15:16:24.891111+0000","last_became_peered":"2024-04-15T15:16:24.891111+0000","last_unstale":"2024-04-15T15:16:26.884867+0000","last_undegraded":"2024-04-15T15:16:26.884867+0000","last_fullsized":"2024-04-15T15:16:26.884867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185587+0000","last_change":"2024-04-15T15:16:25.892154+0000","last_active":"2024-04-15T15:16:27.185587+0000","last_peered":"2024-04-15T15:16:27.185587+0000","last_clean":"2024-04-15T15:16:27.185587+0000","last_became_active":"2024-04-15T15:16:25.891901+0000","last_became_peered":"2024-04-15T15:16:25.891901+0000","last_unstale":"2024-04-15T15:16:27.185587+0000","last_undegraded":"2024-04-15T15:16:27.185587+0000","last_fullsized":"2024-04-15T15:16:27.185587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:48:40.576675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883631+0000","last_change":"2024-04-15T15:16:24.887452+0000","last_active":"2024-04-15T15:16:26.883631+0000","last_peered":"2024-04-15T15:16:26.883631+0000","last_clean":"2024-04-15T15:16:26.883631+0000","last_became_active":"2024-04-15T15:16:24.887136+0000","last_became_peered":"2024-04-15T15:16:24.887136+0000","last_unstale":"2024-04-15T15:16:26.883631+0000","last_undegraded":"2024-04-15T15:16:26.883631+0000","last_fullsized":"2024-04-15T15:16:26.883631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:24.817169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906655+0000","last_change":"2024-04-15T15:16:25.901054+0000","last_active":"2024-04-15T15:16:25.906655+0000","last_peered":"2024-04-15T15:16:25.906655+0000","last_clean":"2024-04-15T15:16:25.906655+0000","last_became_active":"2024-04-15T15:16:25.900784+0000","last_became_peered":"2024-04-15T15:16:25.900784+0000","last_unstale":"2024-04-15T15:16:25.906655+0000","last_undegraded":"2024-04-15T15:16:25.906655+0000","last_fullsized":"2024-04-15T15:16:25.906655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185529+0000","last_change":"2024-04-15T15:16:24.891549+0000","last_active":"2024-04-15T15:16:27.185529+0000","last_peered":"2024-04-15T15:16:27.185529+0000","last_clean":"2024-04-15T15:16:27.185529+0000","last_became_active":"2024-04-15T15:16:24.891268+0000","last_became_peered":"2024-04-15T15:16:24.891268+0000","last_unstale":"2024-04-15T15:16:27.185529+0000","last_undegraded":"2024-04-15T15:16:27.185529+0000","last_fullsized":"2024-04-15T15:16:27.185529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185354+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:16:27.185354+0000","last_peered":"2024-04-15T15:16:27.185354+0000","last_clean":"2024-04-15T15:16:27.185354+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:16:27.185354+0000","last_undegraded":"2024-04-15T15:16:27.185354+0000","last_fullsized":"2024-04-15T15:16:27.185354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1060864,"data_stored":786469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":189373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":778129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:30.601 INFO:tasks.ceph.mds.d.smithi096.stderr:2024-04-15T15:16:30.599+0000 7f3973df2640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:16:33.301 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:16:33.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.416+0000 7febfa02c640 1 Processor -- start 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.416+0000 7febfa02c640 1 -- start start 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febfa02c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febf40d0690 0x7febf40d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febfa02c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7febf40ae280 0x7febf40ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febfa02c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf40d0fb0 con 0x7febf40b3160 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febfa02c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7febf415bda0 con 0x7febf40d0690 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febfa02c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7febf415cb50 con 0x7febf40ae280 2024-04-15T15:16:33.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febf40d0690 0x7febf40d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7febf40d0690 0x7febf40d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47334/0 (socket says 172.21.15.70:47334) 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 -- 172.21.15.70:0/3895457986 learned_addr learned my addr 172.21.15.70:0/3895457986 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf3fff640 1 --2- 172.21.15.70:0/3895457986 >> v2:172.21.15.96:3300/0 conn(0x7febf40ae280 0x7febf40ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 -- 172.21.15.70:0/3895457986 >> v2:172.21.15.96:3300/0 conn(0x7febf40ae280 msgr2=0x7febf40ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 --2- 172.21.15.70:0/3895457986 >> v2:172.21.15.96:3300/0 conn(0x7febf40ae280 0x7febf40ae710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 -- 172.21.15.70:0/3895457986 >> v1:172.21.15.96:6789/0 conn(0x7febf40b3160 legacy=0x7febf415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.417+0000 7febf37fe640 1 -- 172.21.15.70:0/3895457986 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf415bfd0 con 0x7febf40d0690 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.418+0000 7febf37fe640 1 --2- 172.21.15.70:0/3895457986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf40d0a70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7febf406ffe0 tx=0x7febe402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4e26e4a87e6b75b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.418+0000 7febf27fc640 1 -- 172.21.15.70:0/3895457986 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4034070 con 0x7febf40d0690 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.418+0000 7febf27fc640 1 -- 172.21.15.70:0/3895457986 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe4030860 con 0x7febf40d0690 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.418+0000 7febf27fc640 1 -- 172.21.15.70:0/3895457986 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4030c30 con 0x7febf40d0690 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 -- 172.21.15.70:0/3895457986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 msgr2=0x7febf40d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 --2- 172.21.15.70:0/3895457986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf40d0a70 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7febf406ffe0 tx=0x7febe402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 -- 172.21.15.70:0/3895457986 shutdown_connections 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 --2- 172.21.15.70:0/3895457986 >> v2:172.21.15.96:3300/0 conn(0x7febf40ae280 0x7febf40ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 --2- 172.21.15.70:0/3895457986 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf40d0a70 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.426 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 -- 172.21.15.70:0/3895457986 >> 172.21.15.70:0/3895457986 conn(0x7febf406be00 msgr2=0x7febf406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 -- 172.21.15.70:0/3895457986 shutdown_connections 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.419+0000 7febfa02c640 1 -- 172.21.15.70:0/3895457986 wait complete. 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.420+0000 7febfa02c640 1 Processor -- start 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.420+0000 7febfa02c640 1 -- start start 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febf40ae280 0x7febf4147550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febf40b3160 0x7febf4156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7febf415c870 con 0x7febf40d0690 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7febf415c590 con 0x7febf40ae280 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febfa02c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7febf40b4fb0 con 0x7febf40b3160 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47344/0 (socket says 172.21.15.70:47344) 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 -- 172.21.15.70:0/3406604736 learned_addr learned my addr 172.21.15.70:0/3406604736 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:33.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf37fe640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febf40ae280 0x7febf4147550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf2ffd640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febf40b3160 0x7febf4156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 -- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febf40b3160 msgr2=0x7febf4156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febf40b3160 0x7febf4156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 -- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febf40ae280 msgr2=0x7febf4147550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febf40ae280 0x7febf4147550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.421+0000 7febf3fff640 1 -- 172.21.15.70:0/3406604736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf4147a90 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febf3fff640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7febec007e90 tx=0x7febec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec014420 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febec019070 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf4154a90 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec014800 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.422+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf4157b30 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.423+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf40689e0 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.423+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7febec0149a0 con 0x7febf40d0690 2024-04-15T15:16:33.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.424+0000 7febdbfff640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 0x7febe006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.424+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7febec08c510 con 0x7febf40d0690 2024-04-15T15:16:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.424+0000 7febf37fe640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 0x7febe006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.424+0000 7febf37fe640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 0x7febe006edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7febe4036000 tx=0x7febe403b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.428+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7febec05a7b0 con 0x7febf40d0690 2024-04-15T15:16:33.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.598+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7febf40739e0 con 0x7febe006c8f0 2024-04-15T15:16:33.606 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.604+0000 7febdbfff640 1 -- 172.21.15.70:0/3406604736 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251925 (secure 0 0 0) 0x7febf40739e0 con 0x7febe006c8f0 2024-04-15T15:16:33.609 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:33.610 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:16:33.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 msgr2=0x7febe006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 0x7febe006edb0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7febe4036000 tx=0x7febe403b040 comp rx=0 tx=0).stop 2024-04-15T15:16:33.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 msgr2=0x7febf4157220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.612 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7febec007e90 tx=0x7febec00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 shutdown_connections 2024-04-15T15:16:33.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7febe006c8f0 0x7febe006edb0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7febf40d0690 0x7febf4157220 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7febf40b3160 0x7febf4156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 --2- 172.21.15.70:0/3406604736 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7febf40ae280 0x7febf4147550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 >> 172.21.15.70:0/3406604736 conn(0x7febf406be00 msgr2=0x7febf406dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 shutdown_connections 2024-04-15T15:16:33.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.607+0000 7febfa02c640 1 -- 172.21.15.70:0/3406604736 wait complete. 2024-04-15T15:16:33.622 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-04-15T15:16:32.085882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":137,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237476,"kb_used_data":4660,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737244,"statfs":{"total":773094113280,"available":772850937856,"internally_reserved":0,"allocated":4771840,"data_stored":2694064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.213639"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884133+0000","last_change":"2024-04-15T15:16:25.883054+0000","last_active":"2024-04-15T15:16:26.884133+0000","last_peered":"2024-04-15T15:16:26.884133+0000","last_clean":"2024-04-15T15:16:26.884133+0000","last_became_active":"2024-04-15T15:16:25.882744+0000","last_became_peered":"2024-04-15T15:16:25.882744+0000","last_unstale":"2024-04-15T15:16:26.884133+0000","last_undegraded":"2024-04-15T15:16:26.884133+0000","last_fullsized":"2024-04-15T15:16:26.884133+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:06:55.204459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883810+0000","last_change":"2024-04-15T15:16:25.888729+0000","last_active":"2024-04-15T15:16:26.883810+0000","last_peered":"2024-04-15T15:16:26.883810+0000","last_clean":"2024-04-15T15:16:26.883810+0000","last_became_active":"2024-04-15T15:16:25.888450+0000","last_became_peered":"2024-04-15T15:16:25.888450+0000","last_unstale":"2024-04-15T15:16:26.883810+0000","last_undegraded":"2024-04-15T15:16:26.883810+0000","last_fullsized":"2024-04-15T15:16:26.883810+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902977+0000","last_change":"2024-04-15T15:16:25.902269+0000","last_active":"2024-04-15T15:16:25.902977+0000","last_peered":"2024-04-15T15:16:25.902977+0000","last_clean":"2024-04-15T15:16:25.902977+0000","last_became_active":"2024-04-15T15:16:25.901972+0000","last_became_peered":"2024-04-15T15:16:25.901972+0000","last_unstale":"2024-04-15T15:16:25.902977+0000","last_undegraded":"2024-04-15T15:16:25.902977+0000","last_fullsized":"2024-04-15T15:16:25.902977+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898887+0000","last_change":"2024-04-15T15:16:25.897033+0000","last_active":"2024-04-15T15:16:25.898887+0000","last_peered":"2024-04-15T15:16:25.898887+0000","last_clean":"2024-04-15T15:16:25.898887+0000","last_became_active":"2024-04-15T15:16:25.896432+0000","last_became_peered":"2024-04-15T15:16:25.896432+0000","last_unstale":"2024-04-15T15:16:25.898887+0000","last_undegraded":"2024-04-15T15:16:25.898887+0000","last_fullsized":"2024-04-15T15:16:25.898887+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:19.565523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891565+0000","last_change":"2024-04-15T15:16:25.889069+0000","last_active":"2024-04-15T15:16:25.891565+0000","last_peered":"2024-04-15T15:16:25.891565+0000","last_clean":"2024-04-15T15:16:25.891565+0000","last_became_active":"2024-04-15T15:16:25.888793+0000","last_became_peered":"2024-04-15T15:16:25.888793+0000","last_unstale":"2024-04-15T15:16:25.891565+0000","last_undegraded":"2024-04-15T15:16:25.891565+0000","last_fullsized":"2024-04-15T15:16:25.891565+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:48.611396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888862+0000","last_change":"2024-04-15T15:16:25.887072+0000","last_active":"2024-04-15T15:16:25.888862+0000","last_peered":"2024-04-15T15:16:25.888862+0000","last_clean":"2024-04-15T15:16:25.888862+0000","last_became_active":"2024-04-15T15:16:25.886821+0000","last_became_peered":"2024-04-15T15:16:25.886821+0000","last_unstale":"2024-04-15T15:16:25.888862+0000","last_undegraded":"2024-04-15T15:16:25.888862+0000","last_fullsized":"2024-04-15T15:16:25.888862+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904107+0000","last_change":"2024-04-15T15:16:25.900378+0000","last_active":"2024-04-15T15:16:25.904107+0000","last_peered":"2024-04-15T15:16:25.904107+0000","last_clean":"2024-04-15T15:16:25.904107+0000","last_became_active":"2024-04-15T15:16:25.900071+0000","last_became_peered":"2024-04-15T15:16:25.900071+0000","last_unstale":"2024-04-15T15:16:25.904107+0000","last_undegraded":"2024-04-15T15:16:25.904107+0000","last_fullsized":"2024-04-15T15:16:25.904107+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:43.288974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883618+0000","last_change":"2024-04-15T15:16:25.893507+0000","last_active":"2024-04-15T15:16:26.883618+0000","last_peered":"2024-04-15T15:16:26.883618+0000","last_clean":"2024-04-15T15:16:26.883618+0000","last_became_active":"2024-04-15T15:16:25.893005+0000","last_became_peered":"2024-04-15T15:16:25.893005+0000","last_unstale":"2024-04-15T15:16:26.883618+0000","last_undegraded":"2024-04-15T15:16:26.883618+0000","last_fullsized":"2024-04-15T15:16:26.883618+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.895340+0000","last_change":"2024-04-15T15:16:25.894870+0000","last_active":"2024-04-15T15:16:25.895340+0000","last_peered":"2024-04-15T15:16:25.895340+0000","last_clean":"2024-04-15T15:16:25.895340+0000","last_became_active":"2024-04-15T15:16:25.894343+0000","last_became_peered":"2024-04-15T15:16:25.894343+0000","last_unstale":"2024-04-15T15:16:25.895340+0000","last_undegraded":"2024-04-15T15:16:25.895340+0000","last_fullsized":"2024-04-15T15:16:25.895340+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904248+0000","last_change":"2024-04-15T15:16:25.900279+0000","last_active":"2024-04-15T15:16:25.904248+0000","last_peered":"2024-04-15T15:16:25.904248+0000","last_clean":"2024-04-15T15:16:25.904248+0000","last_became_active":"2024-04-15T15:16:25.899969+0000","last_became_peered":"2024-04-15T15:16:25.899969+0000","last_unstale":"2024-04-15T15:16:25.904248+0000","last_undegraded":"2024-04-15T15:16:25.904248+0000","last_fullsized":"2024-04-15T15:16:25.904248+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:19:11.196336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911723+0000","last_change":"2024-04-15T15:16:25.904761+0000","last_active":"2024-04-15T15:16:25.911723+0000","last_peered":"2024-04-15T15:16:25.911723+0000","last_clean":"2024-04-15T15:16:25.911723+0000","last_became_active":"2024-04-15T15:16:25.904477+0000","last_became_peered":"2024-04-15T15:16:25.904477+0000","last_unstale":"2024-04-15T15:16:25.911723+0000","last_undegraded":"2024-04-15T15:16:25.911723+0000","last_fullsized":"2024-04-15T15:16:25.911723+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884502+0000","last_change":"2024-04-15T15:16:25.882646+0000","last_active":"2024-04-15T15:16:25.884502+0000","last_peered":"2024-04-15T15:16:25.884502+0000","last_clean":"2024-04-15T15:16:25.884502+0000","last_became_active":"2024-04-15T15:16:25.882375+0000","last_became_peered":"2024-04-15T15:16:25.882375+0000","last_unstale":"2024-04-15T15:16:25.884502+0000","last_undegraded":"2024-04-15T15:16:25.884502+0000","last_fullsized":"2024-04-15T15:16:25.884502+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906634+0000","last_change":"2024-04-15T15:16:25.901003+0000","last_active":"2024-04-15T15:16:25.906634+0000","last_peered":"2024-04-15T15:16:25.906634+0000","last_clean":"2024-04-15T15:16:25.906634+0000","last_became_active":"2024-04-15T15:16:25.900729+0000","last_became_peered":"2024-04-15T15:16:25.900729+0000","last_unstale":"2024-04-15T15:16:25.906634+0000","last_undegraded":"2024-04-15T15:16:25.906634+0000","last_fullsized":"2024-04-15T15:16:25.906634+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:28:34.769819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885365+0000","last_change":"2024-04-15T15:16:25.899260+0000","last_active":"2024-04-15T15:16:26.885365+0000","last_peered":"2024-04-15T15:16:26.885365+0000","last_clean":"2024-04-15T15:16:26.885365+0000","last_became_active":"2024-04-15T15:16:25.898979+0000","last_became_peered":"2024-04-15T15:16:25.898979+0000","last_unstale":"2024-04-15T15:16:26.885365+0000","last_undegraded":"2024-04-15T15:16:26.885365+0000","last_fullsized":"2024-04-15T15:16:26.885365+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904064+0000","last_change":"2024-04-15T15:16:25.899744+0000","last_active":"2024-04-15T15:16:25.904064+0000","last_peered":"2024-04-15T15:16:25.904064+0000","last_clean":"2024-04-15T15:16:25.904064+0000","last_became_active":"2024-04-15T15:16:25.899472+0000","last_became_peered":"2024-04-15T15:16:25.899472+0000","last_unstale":"2024-04-15T15:16:25.904064+0000","last_undegraded":"2024-04-15T15:16:25.904064+0000","last_fullsized":"2024-04-15T15:16:25.904064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185213+0000","last_change":"2024-04-15T15:16:25.887925+0000","last_active":"2024-04-15T15:16:27.185213+0000","last_peered":"2024-04-15T15:16:27.185213+0000","last_clean":"2024-04-15T15:16:27.185213+0000","last_became_active":"2024-04-15T15:16:25.887628+0000","last_became_peered":"2024-04-15T15:16:25.887628+0000","last_unstale":"2024-04-15T15:16:27.185213+0000","last_undegraded":"2024-04-15T15:16:27.185213+0000","last_fullsized":"2024-04-15T15:16:27.185213+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185828+0000","last_change":"2024-04-15T15:16:25.902349+0000","last_active":"2024-04-15T15:16:27.185828+0000","last_peered":"2024-04-15T15:16:27.185828+0000","last_clean":"2024-04-15T15:16:27.185828+0000","last_became_active":"2024-04-15T15:16:25.901988+0000","last_became_peered":"2024-04-15T15:16:25.901988+0000","last_unstale":"2024-04-15T15:16:27.185828+0000","last_undegraded":"2024-04-15T15:16:27.185828+0000","last_fullsized":"2024-04-15T15:16:27.185828+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T21:55:37.806480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885281+0000","last_change":"2024-04-15T15:16:25.892123+0000","last_active":"2024-04-15T15:16:26.885281+0000","last_peered":"2024-04-15T15:16:26.885281+0000","last_clean":"2024-04-15T15:16:26.885281+0000","last_became_active":"2024-04-15T15:16:25.891853+0000","last_became_peered":"2024-04-15T15:16:25.891853+0000","last_unstale":"2024-04-15T15:16:26.885281+0000","last_undegraded":"2024-04-15T15:16:26.885281+0000","last_fullsized":"2024-04-15T15:16:26.885281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.884688+0000","last_change":"2024-04-15T15:16:25.880039+0000","last_active":"2024-04-15T15:16:25.884688+0000","last_peered":"2024-04-15T15:16:25.884688+0000","last_clean":"2024-04-15T15:16:25.884688+0000","last_became_active":"2024-04-15T15:16:25.879753+0000","last_became_peered":"2024-04-15T15:16:25.879753+0000","last_unstale":"2024-04-15T15:16:25.884688+0000","last_undegraded":"2024-04-15T15:16:25.884688+0000","last_fullsized":"2024-04-15T15:16:25.884688+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:58:15.988048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885269+0000","last_change":"2024-04-15T15:16:25.897291+0000","last_active":"2024-04-15T15:16:26.885269+0000","last_peered":"2024-04-15T15:16:26.885269+0000","last_clean":"2024-04-15T15:16:26.885269+0000","last_became_active":"2024-04-15T15:16:25.896977+0000","last_became_peered":"2024-04-15T15:16:25.896977+0000","last_unstale":"2024-04-15T15:16:26.885269+0000","last_undegraded":"2024-04-15T15:16:26.885269+0000","last_fullsized":"2024-04-15T15:16:26.885269+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901153+0000","last_change":"2024-04-15T15:16:25.889177+0000","last_active":"2024-04-15T15:16:25.901153+0000","last_peered":"2024-04-15T15:16:25.901153+0000","last_clean":"2024-04-15T15:16:25.901153+0000","last_became_active":"2024-04-15T15:16:25.888843+0000","last_became_peered":"2024-04-15T15:16:25.888843+0000","last_unstale":"2024-04-15T15:16:25.901153+0000","last_undegraded":"2024-04-15T15:16:25.901153+0000","last_fullsized":"2024-04-15T15:16:25.901153+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885679+0000","last_change":"2024-04-15T15:16:25.892453+0000","last_active":"2024-04-15T15:16:26.885679+0000","last_peered":"2024-04-15T15:16:26.885679+0000","last_clean":"2024-04-15T15:16:26.885679+0000","last_became_active":"2024-04-15T15:16:25.892168+0000","last_became_peered":"2024-04-15T15:16:25.892168+0000","last_unstale":"2024-04-15T15:16:26.885679+0000","last_undegraded":"2024-04-15T15:16:26.885679+0000","last_fullsized":"2024-04-15T15:16:26.885679+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185288+0000","last_change":"2024-04-15T15:16:25.893780+0000","last_active":"2024-04-15T15:16:27.185288+0000","last_peered":"2024-04-15T15:16:27.185288+0000","last_clean":"2024-04-15T15:16:27.185288+0000","last_became_active":"2024-04-15T15:16:25.893529+0000","last_became_peered":"2024-04-15T15:16:25.893529+0000","last_unstale":"2024-04-15T15:16:27.185288+0000","last_undegraded":"2024-04-15T15:16:27.185288+0000","last_fullsized":"2024-04-15T15:16:27.185288+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891390+0000","last_change":"2024-04-15T15:16:25.885032+0000","last_active":"2024-04-15T15:16:25.891390+0000","last_peered":"2024-04-15T15:16:25.891390+0000","last_clean":"2024-04-15T15:16:25.891390+0000","last_became_active":"2024-04-15T15:16:25.884397+0000","last_became_peered":"2024-04-15T15:16:25.884397+0000","last_unstale":"2024-04-15T15:16:25.891390+0000","last_undegraded":"2024-04-15T15:16:25.891390+0000","last_fullsized":"2024-04-15T15:16:25.891390+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:31.155865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885450+0000","last_change":"2024-04-15T15:16:25.882933+0000","last_active":"2024-04-15T15:16:26.885450+0000","last_peered":"2024-04-15T15:16:26.885450+0000","last_clean":"2024-04-15T15:16:26.885450+0000","last_became_active":"2024-04-15T15:16:25.882643+0000","last_became_peered":"2024-04-15T15:16:25.882643+0000","last_unstale":"2024-04-15T15:16:26.885450+0000","last_undegraded":"2024-04-15T15:16:26.885450+0000","last_fullsized":"2024-04-15T15:16:26.885450+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891210+0000","last_change":"2024-04-15T15:16:25.890339+0000","last_active":"2024-04-15T15:16:25.891210+0000","last_peered":"2024-04-15T15:16:25.891210+0000","last_clean":"2024-04-15T15:16:25.891210+0000","last_became_active":"2024-04-15T15:16:25.889922+0000","last_became_peered":"2024-04-15T15:16:25.889922+0000","last_unstale":"2024-04-15T15:16:25.891210+0000","last_undegraded":"2024-04-15T15:16:25.891210+0000","last_fullsized":"2024-04-15T15:16:25.891210+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885230+0000","last_change":"2024-04-15T15:16:25.890968+0000","last_active":"2024-04-15T15:16:26.885230+0000","last_peered":"2024-04-15T15:16:26.885230+0000","last_clean":"2024-04-15T15:16:26.885230+0000","last_became_active":"2024-04-15T15:16:25.890686+0000","last_became_peered":"2024-04-15T15:16:25.890686+0000","last_unstale":"2024-04-15T15:16:26.885230+0000","last_undegraded":"2024-04-15T15:16:26.885230+0000","last_fullsized":"2024-04-15T15:16:26.885230+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:16.132097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904156+0000","last_change":"2024-04-15T15:16:25.900260+0000","last_active":"2024-04-15T15:16:25.904156+0000","last_peered":"2024-04-15T15:16:25.904156+0000","last_clean":"2024-04-15T15:16:25.904156+0000","last_became_active":"2024-04-15T15:16:25.899948+0000","last_became_peered":"2024-04-15T15:16:25.899948+0000","last_unstale":"2024-04-15T15:16:25.904156+0000","last_undegraded":"2024-04-15T15:16:25.904156+0000","last_fullsized":"2024-04-15T15:16:25.904156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884989+0000","last_change":"2024-04-15T15:16:25.884088+0000","last_active":"2024-04-15T15:16:26.884989+0000","last_peered":"2024-04-15T15:16:26.884989+0000","last_clean":"2024-04-15T15:16:26.884989+0000","last_became_active":"2024-04-15T15:16:25.883804+0000","last_became_peered":"2024-04-15T15:16:25.883804+0000","last_unstale":"2024-04-15T15:16:26.884989+0000","last_undegraded":"2024-04-15T15:16:26.884989+0000","last_fullsized":"2024-04-15T15:16:26.884989+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885622+0000","last_change":"2024-04-15T15:16:25.879663+0000","last_active":"2024-04-15T15:16:26.885622+0000","last_peered":"2024-04-15T15:16:26.885622+0000","last_clean":"2024-04-15T15:16:26.885622+0000","last_became_active":"2024-04-15T15:16:25.879385+0000","last_became_peered":"2024-04-15T15:16:25.879385+0000","last_unstale":"2024-04-15T15:16:26.885622+0000","last_undegraded":"2024-04-15T15:16:26.885622+0000","last_fullsized":"2024-04-15T15:16:26.885622+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185437+0000","last_change":"2024-04-15T15:16:25.893793+0000","last_active":"2024-04-15T15:16:27.185437+0000","last_peered":"2024-04-15T15:16:27.185437+0000","last_clean":"2024-04-15T15:16:27.185437+0000","last_became_active":"2024-04-15T15:16:25.893556+0000","last_became_peered":"2024-04-15T15:16:25.893556+0000","last_unstale":"2024-04-15T15:16:27.185437+0000","last_undegraded":"2024-04-15T15:16:27.185437+0000","last_fullsized":"2024-04-15T15:16:27.185437+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:10:46.054407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888033+0000","last_change":"2024-04-15T15:16:25.886972+0000","last_active":"2024-04-15T15:16:25.888033+0000","last_peered":"2024-04-15T15:16:25.888033+0000","last_clean":"2024-04-15T15:16:25.888033+0000","last_became_active":"2024-04-15T15:16:25.886718+0000","last_became_peered":"2024-04-15T15:16:25.886718+0000","last_unstale":"2024-04-15T15:16:25.888033+0000","last_undegraded":"2024-04-15T15:16:25.888033+0000","last_fullsized":"2024-04-15T15:16:25.888033+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185663+0000","last_change":"2024-04-15T15:16:25.896046+0000","last_active":"2024-04-15T15:16:27.185663+0000","last_peered":"2024-04-15T15:16:27.185663+0000","last_clean":"2024-04-15T15:16:27.185663+0000","last_became_active":"2024-04-15T15:16:25.895768+0000","last_became_peered":"2024-04-15T15:16:25.895768+0000","last_unstale":"2024-04-15T15:16:27.185663+0000","last_undegraded":"2024-04-15T15:16:27.185663+0000","last_fullsized":"2024-04-15T15:16:27.185663+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885577+0000","last_change":"2024-04-15T15:16:24.886178+0000","last_active":"2024-04-15T15:16:26.885577+0000","last_peered":"2024-04-15T15:16:26.885577+0000","last_clean":"2024-04-15T15:16:26.885577+0000","last_became_active":"2024-04-15T15:16:24.885910+0000","last_became_peered":"2024-04-15T15:16:24.885910+0000","last_unstale":"2024-04-15T15:16:26.885577+0000","last_undegraded":"2024-04-15T15:16:26.885577+0000","last_fullsized":"2024-04-15T15:16:26.885577+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903528+0000","last_change":"2024-04-15T15:16:25.902820+0000","last_active":"2024-04-15T15:16:25.903528+0000","last_peered":"2024-04-15T15:16:25.903528+0000","last_clean":"2024-04-15T15:16:25.903528+0000","last_became_active":"2024-04-15T15:16:25.902552+0000","last_became_peered":"2024-04-15T15:16:25.902552+0000","last_unstale":"2024-04-15T15:16:25.903528+0000","last_undegraded":"2024-04-15T15:16:25.903528+0000","last_fullsized":"2024-04-15T15:16:25.903528+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898338+0000","last_change":"2024-04-15T15:16:24.886331+0000","last_active":"2024-04-15T15:16:25.898338+0000","last_peered":"2024-04-15T15:16:25.898338+0000","last_clean":"2024-04-15T15:16:25.898338+0000","last_became_active":"2024-04-15T15:16:24.886052+0000","last_became_peered":"2024-04-15T15:16:24.886052+0000","last_unstale":"2024-04-15T15:16:25.898338+0000","last_undegraded":"2024-04-15T15:16:25.898338+0000","last_fullsized":"2024-04-15T15:16:25.898338+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.899883+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.899620+0000","last_became_peered":"2024-04-15T15:16:25.899620+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:29:59.437287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898694+0000","last_change":"2024-04-15T15:16:24.890646+0000","last_active":"2024-04-15T15:16:25.898694+0000","last_peered":"2024-04-15T15:16:25.898694+0000","last_clean":"2024-04-15T15:16:25.898694+0000","last_became_active":"2024-04-15T15:16:24.890362+0000","last_became_peered":"2024-04-15T15:16:24.890362+0000","last_unstale":"2024-04-15T15:16:25.898694+0000","last_undegraded":"2024-04-15T15:16:25.898694+0000","last_fullsized":"2024-04-15T15:16:25.898694+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883847+0000","last_change":"2024-04-15T15:16:25.895423+0000","last_active":"2024-04-15T15:16:26.883847+0000","last_peered":"2024-04-15T15:16:26.883847+0000","last_clean":"2024-04-15T15:16:26.883847+0000","last_became_active":"2024-04-15T15:16:25.889905+0000","last_became_peered":"2024-04-15T15:16:25.889905+0000","last_unstale":"2024-04-15T15:16:26.883847+0000","last_undegraded":"2024-04-15T15:16:26.883847+0000","last_fullsized":"2024-04-15T15:16:26.883847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:50:15.175711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886111+0000","last_change":"2024-04-15T15:16:24.887352+0000","last_active":"2024-04-15T15:16:26.886111+0000","last_peered":"2024-04-15T15:16:26.886111+0000","last_clean":"2024-04-15T15:16:26.886111+0000","last_became_active":"2024-04-15T15:16:24.887054+0000","last_became_peered":"2024-04-15T15:16:24.887054+0000","last_unstale":"2024-04-15T15:16:26.886111+0000","last_undegraded":"2024-04-15T15:16:26.886111+0000","last_fullsized":"2024-04-15T15:16:26.886111+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.882206+0000","last_change":"2024-04-15T15:16:25.881247+0000","last_active":"2024-04-15T15:16:25.882206+0000","last_peered":"2024-04-15T15:16:25.882206+0000","last_clean":"2024-04-15T15:16:25.882206+0000","last_became_active":"2024-04-15T15:16:25.880981+0000","last_became_peered":"2024-04-15T15:16:25.880981+0000","last_unstale":"2024-04-15T15:16:25.882206+0000","last_undegraded":"2024-04-15T15:16:25.882206+0000","last_fullsized":"2024-04-15T15:16:25.882206+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:43:00.918821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903854+0000","last_change":"2024-04-15T15:16:24.891022+0000","last_active":"2024-04-15T15:16:25.903854+0000","last_peered":"2024-04-15T15:16:25.903854+0000","last_clean":"2024-04-15T15:16:25.903854+0000","last_became_active":"2024-04-15T15:16:24.890770+0000","last_became_peered":"2024-04-15T15:16:24.890770+0000","last_unstale":"2024-04-15T15:16:25.903854+0000","last_undegraded":"2024-04-15T15:16:25.903854+0000","last_fullsized":"2024-04-15T15:16:25.903854+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.901334+0000","last_change":"2024-04-15T15:16:25.893573+0000","last_active":"2024-04-15T15:16:25.901334+0000","last_peered":"2024-04-15T15:16:25.901334+0000","last_clean":"2024-04-15T15:16:25.901334+0000","last_became_active":"2024-04-15T15:16:25.893321+0000","last_became_peered":"2024-04-15T15:16:25.893321+0000","last_unstale":"2024-04-15T15:16:25.901334+0000","last_undegraded":"2024-04-15T15:16:25.901334+0000","last_fullsized":"2024-04-15T15:16:25.901334+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185361+0000","last_change":"2024-04-15T15:16:24.889597+0000","last_active":"2024-04-15T15:16:27.185361+0000","last_peered":"2024-04-15T15:16:27.185361+0000","last_clean":"2024-04-15T15:16:27.185361+0000","last_became_active":"2024-04-15T15:16:24.889176+0000","last_became_peered":"2024-04-15T15:16:24.889176+0000","last_unstale":"2024-04-15T15:16:27.185361+0000","last_undegraded":"2024-04-15T15:16:27.185361+0000","last_fullsized":"2024-04-15T15:16:27.185361+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885832+0000","last_change":"2024-04-15T15:16:25.904897+0000","last_active":"2024-04-15T15:16:26.885832+0000","last_peered":"2024-04-15T15:16:26.885832+0000","last_clean":"2024-04-15T15:16:26.885832+0000","last_became_active":"2024-04-15T15:16:25.904610+0000","last_became_peered":"2024-04-15T15:16:25.904610+0000","last_unstale":"2024-04-15T15:16:26.885832+0000","last_undegraded":"2024-04-15T15:16:26.885832+0000","last_fullsized":"2024-04-15T15:16:26.885832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876235+0000","last_change":"2024-04-15T15:16:24.891428+0000","last_active":"2024-04-15T15:16:25.876235+0000","last_peered":"2024-04-15T15:16:25.876235+0000","last_clean":"2024-04-15T15:16:25.876235+0000","last_became_active":"2024-04-15T15:16:24.891151+0000","last_became_peered":"2024-04-15T15:16:24.891151+0000","last_unstale":"2024-04-15T15:16:25.876235+0000","last_undegraded":"2024-04-15T15:16:25.876235+0000","last_fullsized":"2024-04-15T15:16:25.876235+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:56:54.756234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883767+0000","last_change":"2024-04-15T15:16:25.898140+0000","last_active":"2024-04-15T15:16:26.883767+0000","last_peered":"2024-04-15T15:16:26.883767+0000","last_clean":"2024-04-15T15:16:26.883767+0000","last_became_active":"2024-04-15T15:16:25.897492+0000","last_became_peered":"2024-04-15T15:16:25.897492+0000","last_unstale":"2024-04-15T15:16:26.883767+0000","last_undegraded":"2024-04-15T15:16:26.883767+0000","last_fullsized":"2024-04-15T15:16:26.883767+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:13:20.475225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884775+0000","last_change":"2024-04-15T15:16:24.884571+0000","last_active":"2024-04-15T15:16:26.884775+0000","last_peered":"2024-04-15T15:16:26.884775+0000","last_clean":"2024-04-15T15:16:26.884775+0000","last_became_active":"2024-04-15T15:16:24.884274+0000","last_became_peered":"2024-04-15T15:16:24.884274+0000","last_unstale":"2024-04-15T15:16:26.884775+0000","last_undegraded":"2024-04-15T15:16:26.884775+0000","last_fullsized":"2024-04-15T15:16:26.884775+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:41:47.609097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.899098+0000","last_change":"2024-04-15T15:16:25.898346+0000","last_active":"2024-04-15T15:16:25.899098+0000","last_peered":"2024-04-15T15:16:25.899098+0000","last_clean":"2024-04-15T15:16:25.899098+0000","last_became_active":"2024-04-15T15:16:25.897145+0000","last_became_peered":"2024-04-15T15:16:25.897145+0000","last_unstale":"2024-04-15T15:16:25.899098+0000","last_undegraded":"2024-04-15T15:16:25.899098+0000","last_fullsized":"2024-04-15T15:16:25.899098+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:54.775491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884135+0000","last_change":"2024-04-15T15:16:24.888936+0000","last_active":"2024-04-15T15:16:26.884135+0000","last_peered":"2024-04-15T15:16:26.884135+0000","last_clean":"2024-04-15T15:16:26.884135+0000","last_became_active":"2024-04-15T15:16:24.888050+0000","last_became_peered":"2024-04-15T15:16:24.888050+0000","last_unstale":"2024-04-15T15:16:26.884135+0000","last_undegraded":"2024-04-15T15:16:26.884135+0000","last_fullsized":"2024-04-15T15:16:26.884135+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185633+0000","last_change":"2024-04-15T15:16:25.899963+0000","last_active":"2024-04-15T15:16:27.185633+0000","last_peered":"2024-04-15T15:16:27.185633+0000","last_clean":"2024-04-15T15:16:27.185633+0000","last_became_active":"2024-04-15T15:16:25.899676+0000","last_became_peered":"2024-04-15T15:16:25.899676+0000","last_unstale":"2024-04-15T15:16:27.185633+0000","last_undegraded":"2024-04-15T15:16:27.185633+0000","last_fullsized":"2024-04-15T15:16:27.185633+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:30:14.060100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.886799+0000","last_change":"2024-04-15T15:16:24.887199+0000","last_active":"2024-04-15T15:16:25.886799+0000","last_peered":"2024-04-15T15:16:25.886799+0000","last_clean":"2024-04-15T15:16:25.886799+0000","last_became_active":"2024-04-15T15:16:24.886945+0000","last_became_peered":"2024-04-15T15:16:24.886945+0000","last_unstale":"2024-04-15T15:16:25.886799+0000","last_undegraded":"2024-04-15T15:16:25.886799+0000","last_fullsized":"2024-04-15T15:16:25.886799+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.893867+0000","last_change":"2024-04-15T15:16:25.892203+0000","last_active":"2024-04-15T15:16:25.893867+0000","last_peered":"2024-04-15T15:16:25.893867+0000","last_clean":"2024-04-15T15:16:25.893867+0000","last_became_active":"2024-04-15T15:16:25.891899+0000","last_became_peered":"2024-04-15T15:16:25.891899+0000","last_unstale":"2024-04-15T15:16:25.893867+0000","last_undegraded":"2024-04-15T15:16:25.893867+0000","last_fullsized":"2024-04-15T15:16:25.893867+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:41.689909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":15,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.904566+0000","last_change":"2024-04-15T15:16:24.890265+0000","last_active":"2024-04-15T15:16:25.904566+0000","last_peered":"2024-04-15T15:16:25.904566+0000","last_clean":"2024-04-15T15:16:25.904566+0000","last_became_active":"2024-04-15T15:16:24.889963+0000","last_became_peered":"2024-04-15T15:16:24.889963+0000","last_unstale":"2024-04-15T15:16:25.904566+0000","last_undegraded":"2024-04-15T15:16:25.904566+0000","last_fullsized":"2024-04-15T15:16:25.904566+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:56:40.687610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886503+0000","last_change":"2024-04-15T15:16:25.893921+0000","last_active":"2024-04-15T15:16:26.886503+0000","last_peered":"2024-04-15T15:16:26.886503+0000","last_clean":"2024-04-15T15:16:26.886503+0000","last_became_active":"2024-04-15T15:16:25.892154+0000","last_became_peered":"2024-04-15T15:16:25.892154+0000","last_unstale":"2024-04-15T15:16:26.886503+0000","last_undegraded":"2024-04-15T15:16:26.886503+0000","last_fullsized":"2024-04-15T15:16:26.886503+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:16.631742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884185+0000","last_change":"2024-04-15T15:16:24.889313+0000","last_active":"2024-04-15T15:16:26.884185+0000","last_peered":"2024-04-15T15:16:26.884185+0000","last_clean":"2024-04-15T15:16:26.884185+0000","last_became_active":"2024-04-15T15:16:24.889037+0000","last_became_peered":"2024-04-15T15:16:24.889037+0000","last_unstale":"2024-04-15T15:16:26.884185+0000","last_undegraded":"2024-04-15T15:16:26.884185+0000","last_fullsized":"2024-04-15T15:16:26.884185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:58.773260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.889667+0000","last_change":"2024-04-15T15:16:25.888305+0000","last_active":"2024-04-15T15:16:25.889667+0000","last_peered":"2024-04-15T15:16:25.889667+0000","last_clean":"2024-04-15T15:16:25.889667+0000","last_became_active":"2024-04-15T15:16:25.888053+0000","last_became_peered":"2024-04-15T15:16:25.888053+0000","last_unstale":"2024-04-15T15:16:25.889667+0000","last_undegraded":"2024-04-15T15:16:25.889667+0000","last_fullsized":"2024-04-15T15:16:25.889667+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:09.237587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884136+0000","last_change":"2024-04-15T15:16:24.882824+0000","last_active":"2024-04-15T15:16:26.884136+0000","last_peered":"2024-04-15T15:16:26.884136+0000","last_clean":"2024-04-15T15:16:26.884136+0000","last_became_active":"2024-04-15T15:16:24.882560+0000","last_became_peered":"2024-04-15T15:16:24.882560+0000","last_unstale":"2024-04-15T15:16:26.884136+0000","last_undegraded":"2024-04-15T15:16:26.884136+0000","last_fullsized":"2024-04-15T15:16:26.884136+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:24:08.235288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883781+0000","last_change":"2024-04-15T15:16:25.883512+0000","last_active":"2024-04-15T15:16:26.883781+0000","last_peered":"2024-04-15T15:16:26.883781+0000","last_clean":"2024-04-15T15:16:26.883781+0000","last_became_active":"2024-04-15T15:16:25.883253+0000","last_became_peered":"2024-04-15T15:16:25.883253+0000","last_unstale":"2024-04-15T15:16:26.883781+0000","last_undegraded":"2024-04-15T15:16:26.883781+0000","last_fullsized":"2024-04-15T15:16:26.883781+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.877572+0000","last_change":"2024-04-15T15:16:24.891194+0000","last_active":"2024-04-15T15:16:25.877572+0000","last_peered":"2024-04-15T15:16:25.877572+0000","last_clean":"2024-04-15T15:16:25.877572+0000","last_became_active":"2024-04-15T15:16:24.890944+0000","last_became_peered":"2024-04-15T15:16:24.890944+0000","last_unstale":"2024-04-15T15:16:25.877572+0000","last_undegraded":"2024-04-15T15:16:25.877572+0000","last_fullsized":"2024-04-15T15:16:25.877572+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185589+0000","last_change":"2024-04-15T15:16:25.893190+0000","last_active":"2024-04-15T15:16:27.185589+0000","last_peered":"2024-04-15T15:16:27.185589+0000","last_clean":"2024-04-15T15:16:27.185589+0000","last_became_active":"2024-04-15T15:16:25.892917+0000","last_became_peered":"2024-04-15T15:16:25.892917+0000","last_unstale":"2024-04-15T15:16:27.185589+0000","last_undegraded":"2024-04-15T15:16:27.185589+0000","last_fullsized":"2024-04-15T15:16:27.185589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T15:43:41.370786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885147+0000","last_change":"2024-04-15T15:16:24.886029+0000","last_active":"2024-04-15T15:16:26.885147+0000","last_peered":"2024-04-15T15:16:26.885147+0000","last_clean":"2024-04-15T15:16:26.885147+0000","last_became_active":"2024-04-15T15:16:24.885730+0000","last_became_peered":"2024-04-15T15:16:24.885730+0000","last_unstale":"2024-04-15T15:16:26.885147+0000","last_undegraded":"2024-04-15T15:16:26.885147+0000","last_fullsized":"2024-04-15T15:16:26.885147+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:36:53.773191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884020+0000","last_change":"2024-04-15T15:16:25.882916+0000","last_active":"2024-04-15T15:16:26.884020+0000","last_peered":"2024-04-15T15:16:26.884020+0000","last_clean":"2024-04-15T15:16:26.884020+0000","last_became_active":"2024-04-15T15:16:25.882630+0000","last_became_peered":"2024-04-15T15:16:25.882630+0000","last_unstale":"2024-04-15T15:16:26.884020+0000","last_undegraded":"2024-04-15T15:16:26.884020+0000","last_fullsized":"2024-04-15T15:16:26.884020+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:26:24.310942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185272+0000","last_change":"2024-04-15T15:16:24.889751+0000","last_active":"2024-04-15T15:16:27.185272+0000","last_peered":"2024-04-15T15:16:27.185272+0000","last_clean":"2024-04-15T15:16:27.185272+0000","last_became_active":"2024-04-15T15:16:24.889428+0000","last_became_peered":"2024-04-15T15:16:24.889428+0000","last_unstale":"2024-04-15T15:16:27.185272+0000","last_undegraded":"2024-04-15T15:16:27.185272+0000","last_fullsized":"2024-04-15T15:16:27.185272+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884117+0000","last_change":"2024-04-15T15:16:25.890929+0000","last_active":"2024-04-15T15:16:26.884117+0000","last_peered":"2024-04-15T15:16:26.884117+0000","last_clean":"2024-04-15T15:16:26.884117+0000","last_became_active":"2024-04-15T15:16:25.890256+0000","last_became_peered":"2024-04-15T15:16:25.890256+0000","last_unstale":"2024-04-15T15:16:26.884117+0000","last_undegraded":"2024-04-15T15:16:26.884117+0000","last_fullsized":"2024-04-15T15:16:26.884117+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:39:11.834293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906758+0000","last_change":"2024-04-15T15:16:24.893507+0000","last_active":"2024-04-15T15:16:25.906758+0000","last_peered":"2024-04-15T15:16:25.906758+0000","last_clean":"2024-04-15T15:16:25.906758+0000","last_became_active":"2024-04-15T15:16:24.893262+0000","last_became_peered":"2024-04-15T15:16:24.893262+0000","last_unstale":"2024-04-15T15:16:25.906758+0000","last_undegraded":"2024-04-15T15:16:25.906758+0000","last_fullsized":"2024-04-15T15:16:25.906758+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:35:15.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.911705+0000","last_change":"2024-04-15T15:16:25.904384+0000","last_active":"2024-04-15T15:16:25.911705+0000","last_peered":"2024-04-15T15:16:25.911705+0000","last_clean":"2024-04-15T15:16:25.911705+0000","last_became_active":"2024-04-15T15:16:25.904095+0000","last_became_peered":"2024-04-15T15:16:25.904095+0000","last_unstale":"2024-04-15T15:16:25.911705+0000","last_undegraded":"2024-04-15T15:16:25.911705+0000","last_fullsized":"2024-04-15T15:16:25.911705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:40:30.677974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886583+0000","last_change":"2024-04-15T15:16:24.891423+0000","last_active":"2024-04-15T15:16:26.886583+0000","last_peered":"2024-04-15T15:16:26.886583+0000","last_clean":"2024-04-15T15:16:26.886583+0000","last_became_active":"2024-04-15T15:16:24.891048+0000","last_became_peered":"2024-04-15T15:16:24.891048+0000","last_unstale":"2024-04-15T15:16:26.886583+0000","last_undegraded":"2024-04-15T15:16:26.886583+0000","last_fullsized":"2024-04-15T15:16:26.886583+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:47:10.367180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.886486+0000","last_change":"2024-04-15T15:16:25.894040+0000","last_active":"2024-04-15T15:16:26.886486+0000","last_peered":"2024-04-15T15:16:26.886486+0000","last_clean":"2024-04-15T15:16:26.886486+0000","last_became_active":"2024-04-15T15:16:25.893781+0000","last_became_peered":"2024-04-15T15:16:25.893781+0000","last_unstale":"2024-04-15T15:16:26.886486+0000","last_undegraded":"2024-04-15T15:16:26.886486+0000","last_fullsized":"2024-04-15T15:16:26.886486+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":13,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.897493+0000","last_change":"2024-04-15T15:16:24.889598+0000","last_active":"2024-04-15T15:16:25.897493+0000","last_peered":"2024-04-15T15:16:25.897493+0000","last_clean":"2024-04-15T15:16:25.897493+0000","last_became_active":"2024-04-15T15:16:24.889298+0000","last_became_peered":"2024-04-15T15:16:24.889298+0000","last_unstale":"2024-04-15T15:16:25.897493+0000","last_undegraded":"2024-04-15T15:16:25.897493+0000","last_fullsized":"2024-04-15T15:16:25.897493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:54:18.385006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891485+0000","last_change":"2024-04-15T15:16:25.884522+0000","last_active":"2024-04-15T15:16:25.891485+0000","last_peered":"2024-04-15T15:16:25.891485+0000","last_clean":"2024-04-15T15:16:25.891485+0000","last_became_active":"2024-04-15T15:16:25.883126+0000","last_became_peered":"2024-04-15T15:16:25.883126+0000","last_unstale":"2024-04-15T15:16:25.891485+0000","last_undegraded":"2024-04-15T15:16:25.891485+0000","last_fullsized":"2024-04-15T15:16:25.891485+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:46:03.969456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876163+0000","last_change":"2024-04-15T15:16:24.890031+0000","last_active":"2024-04-15T15:16:25.876163+0000","last_peered":"2024-04-15T15:16:25.876163+0000","last_clean":"2024-04-15T15:16:25.876163+0000","last_became_active":"2024-04-15T15:16:24.889203+0000","last_became_peered":"2024-04-15T15:16:24.889203+0000","last_unstale":"2024-04-15T15:16:25.876163+0000","last_undegraded":"2024-04-15T15:16:25.876163+0000","last_fullsized":"2024-04-15T15:16:25.876163+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:13:18.919033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185440+0000","last_change":"2024-04-15T15:16:25.894269+0000","last_active":"2024-04-15T15:16:27.185440+0000","last_peered":"2024-04-15T15:16:27.185440+0000","last_clean":"2024-04-15T15:16:27.185440+0000","last_became_active":"2024-04-15T15:16:25.893279+0000","last_became_peered":"2024-04-15T15:16:25.893279+0000","last_unstale":"2024-04-15T15:16:27.185440+0000","last_undegraded":"2024-04-15T15:16:27.185440+0000","last_fullsized":"2024-04-15T15:16:27.185440+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T17:48:22.937069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.891401+0000","last_change":"2024-04-15T15:16:24.877420+0000","last_active":"2024-04-15T15:16:25.891401+0000","last_peered":"2024-04-15T15:16:25.891401+0000","last_clean":"2024-04-15T15:16:25.891401+0000","last_became_active":"2024-04-15T15:16:24.877149+0000","last_became_peered":"2024-04-15T15:16:24.877149+0000","last_unstale":"2024-04-15T15:16:25.891401+0000","last_undegraded":"2024-04-15T15:16:25.891401+0000","last_fullsized":"2024-04-15T15:16:25.891401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:31:48.237807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.890064+0000","last_change":"2024-04-15T15:16:25.887625+0000","last_active":"2024-04-15T15:16:25.890064+0000","last_peered":"2024-04-15T15:16:25.890064+0000","last_clean":"2024-04-15T15:16:25.890064+0000","last_became_active":"2024-04-15T15:16:25.887362+0000","last_became_peered":"2024-04-15T15:16:25.887362+0000","last_unstale":"2024-04-15T15:16:25.890064+0000","last_undegraded":"2024-04-15T15:16:25.890064+0000","last_fullsized":"2024-04-15T15:16:25.890064+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872610+0000","last_change":"2024-04-15T15:16:24.884150+0000","last_active":"2024-04-15T15:16:25.872610+0000","last_peered":"2024-04-15T15:16:25.872610+0000","last_clean":"2024-04-15T15:16:25.872610+0000","last_became_active":"2024-04-15T15:16:24.883908+0000","last_became_peered":"2024-04-15T15:16:24.883908+0000","last_unstale":"2024-04-15T15:16:25.872610+0000","last_undegraded":"2024-04-15T15:16:25.872610+0000","last_fullsized":"2024-04-15T15:16:25.872610+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:58:35.147306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.903908+0000","last_change":"2024-04-15T15:16:25.897701+0000","last_active":"2024-04-15T15:16:25.903908+0000","last_peered":"2024-04-15T15:16:25.903908+0000","last_clean":"2024-04-15T15:16:25.903908+0000","last_became_active":"2024-04-15T15:16:25.896891+0000","last_became_peered":"2024-04-15T15:16:25.896891+0000","last_unstale":"2024-04-15T15:16:25.903908+0000","last_undegraded":"2024-04-15T15:16:25.903908+0000","last_fullsized":"2024-04-15T15:16:25.903908+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T03:02:59.775559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185493+0000","last_change":"2024-04-15T15:16:24.887471+0000","last_active":"2024-04-15T15:16:27.185493+0000","last_peered":"2024-04-15T15:16:27.185493+0000","last_clean":"2024-04-15T15:16:27.185493+0000","last_became_active":"2024-04-15T15:16:24.887165+0000","last_became_peered":"2024-04-15T15:16:24.887165+0000","last_unstale":"2024-04-15T15:16:27.185493+0000","last_undegraded":"2024-04-15T15:16:27.185493+0000","last_fullsized":"2024-04-15T15:16:27.185493+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:32.833482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884030+0000","last_change":"2024-04-15T15:16:25.899465+0000","last_active":"2024-04-15T15:16:26.884030+0000","last_peered":"2024-04-15T15:16:26.884030+0000","last_clean":"2024-04-15T15:16:26.884030+0000","last_became_active":"2024-04-15T15:16:25.899205+0000","last_became_peered":"2024-04-15T15:16:25.899205+0000","last_unstale":"2024-04-15T15:16:26.884030+0000","last_undegraded":"2024-04-15T15:16:26.884030+0000","last_fullsized":"2024-04-15T15:16:26.884030+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:42:16.777997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.872575+0000","last_change":"2024-04-15T15:16:24.882714+0000","last_active":"2024-04-15T15:16:25.872575+0000","last_peered":"2024-04-15T15:16:25.872575+0000","last_clean":"2024-04-15T15:16:25.872575+0000","last_became_active":"2024-04-15T15:16:24.882460+0000","last_became_peered":"2024-04-15T15:16:24.882460+0000","last_unstale":"2024-04-15T15:16:25.872575+0000","last_undegraded":"2024-04-15T15:16:25.872575+0000","last_fullsized":"2024-04-15T15:16:25.872575+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:11.251738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.898832+0000","last_change":"2024-04-15T15:16:25.897148+0000","last_active":"2024-04-15T15:16:25.898832+0000","last_peered":"2024-04-15T15:16:25.898832+0000","last_clean":"2024-04-15T15:16:25.898832+0000","last_became_active":"2024-04-15T15:16:25.896866+0000","last_became_peered":"2024-04-15T15:16:25.896866+0000","last_unstale":"2024-04-15T15:16:25.898832+0000","last_undegraded":"2024-04-15T15:16:25.898832+0000","last_fullsized":"2024-04-15T15:16:25.898832+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":14,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.902029+0000","last_change":"2024-04-15T15:16:24.892956+0000","last_active":"2024-04-15T15:16:25.902029+0000","last_peered":"2024-04-15T15:16:25.902029+0000","last_clean":"2024-04-15T15:16:25.902029+0000","last_became_active":"2024-04-15T15:16:24.892664+0000","last_became_peered":"2024-04-15T15:16:24.892664+0000","last_unstale":"2024-04-15T15:16:25.902029+0000","last_undegraded":"2024-04-15T15:16:25.902029+0000","last_fullsized":"2024-04-15T15:16:25.902029+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:56.013415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185690+0000","last_change":"2024-04-15T15:16:25.899975+0000","last_active":"2024-04-15T15:16:27.185690+0000","last_peered":"2024-04-15T15:16:27.185690+0000","last_clean":"2024-04-15T15:16:27.185690+0000","last_became_active":"2024-04-15T15:16:25.899691+0000","last_became_peered":"2024-04-15T15:16:25.899691+0000","last_unstale":"2024-04-15T15:16:27.185690+0000","last_undegraded":"2024-04-15T15:16:27.185690+0000","last_fullsized":"2024-04-15T15:16:27.185690+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:21:46.867523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.873000+0000","last_change":"2024-04-15T15:16:24.890594+0000","last_active":"2024-04-15T15:16:25.873000+0000","last_peered":"2024-04-15T15:16:25.873000+0000","last_clean":"2024-04-15T15:16:25.873000+0000","last_became_active":"2024-04-15T15:16:24.890305+0000","last_became_peered":"2024-04-15T15:16:24.890305+0000","last_unstale":"2024-04-15T15:16:25.873000+0000","last_undegraded":"2024-04-15T15:16:25.873000+0000","last_fullsized":"2024-04-15T15:16:25.873000+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:19:34.555529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884060+0000","last_change":"2024-04-15T15:16:25.893319+0000","last_active":"2024-04-15T15:16:26.884060+0000","last_peered":"2024-04-15T15:16:26.884060+0000","last_clean":"2024-04-15T15:16:26.884060+0000","last_became_active":"2024-04-15T15:16:25.892794+0000","last_became_peered":"2024-04-15T15:16:25.892794+0000","last_unstale":"2024-04-15T15:16:26.884060+0000","last_undegraded":"2024-04-15T15:16:26.884060+0000","last_fullsized":"2024-04-15T15:16:26.884060+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:08:42.664115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.876776+0000","last_change":"2024-04-15T15:16:24.884802+0000","last_active":"2024-04-15T15:16:25.876776+0000","last_peered":"2024-04-15T15:16:25.876776+0000","last_clean":"2024-04-15T15:16:25.876776+0000","last_became_active":"2024-04-15T15:16:24.884519+0000","last_became_peered":"2024-04-15T15:16:24.884519+0000","last_unstale":"2024-04-15T15:16:25.876776+0000","last_undegraded":"2024-04-15T15:16:25.876776+0000","last_fullsized":"2024-04-15T15:16:25.876776+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:50:00.539366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888791+0000","last_change":"2024-04-15T15:16:25.883444+0000","last_active":"2024-04-15T15:16:25.888791+0000","last_peered":"2024-04-15T15:16:25.888791+0000","last_clean":"2024-04-15T15:16:25.888791+0000","last_became_active":"2024-04-15T15:16:25.883191+0000","last_became_peered":"2024-04-15T15:16:25.883191+0000","last_unstale":"2024-04-15T15:16:25.888791+0000","last_undegraded":"2024-04-15T15:16:25.888791+0000","last_fullsized":"2024-04-15T15:16:25.888791+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T01:52:57.015714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.875590+0000","last_change":"2024-04-15T15:16:24.886148+0000","last_active":"2024-04-15T15:16:25.875590+0000","last_peered":"2024-04-15T15:16:25.875590+0000","last_clean":"2024-04-15T15:16:25.875590+0000","last_became_active":"2024-04-15T15:16:24.885865+0000","last_became_peered":"2024-04-15T15:16:24.885865+0000","last_unstale":"2024-04-15T15:16:25.875590+0000","last_undegraded":"2024-04-15T15:16:25.875590+0000","last_fullsized":"2024-04-15T15:16:25.875590+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T00:27:55.455280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.888002+0000","last_change":"2024-04-15T15:16:25.886178+0000","last_active":"2024-04-15T15:16:25.888002+0000","last_peered":"2024-04-15T15:16:25.888002+0000","last_clean":"2024-04-15T15:16:25.888002+0000","last_became_active":"2024-04-15T15:16:25.885915+0000","last_became_peered":"2024-04-15T15:16:25.885915+0000","last_unstale":"2024-04-15T15:16:25.888002+0000","last_undegraded":"2024-04-15T15:16:25.888002+0000","last_fullsized":"2024-04-15T15:16:25.888002+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T23:23:33.835781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885719+0000","last_change":"2024-04-15T15:16:24.883958+0000","last_active":"2024-04-15T15:16:26.885719+0000","last_peered":"2024-04-15T15:16:26.885719+0000","last_clean":"2024-04-15T15:16:26.885719+0000","last_became_active":"2024-04-15T15:16:24.883528+0000","last_became_peered":"2024-04-15T15:16:24.883528+0000","last_unstale":"2024-04-15T15:16:26.885719+0000","last_undegraded":"2024-04-15T15:16:26.885719+0000","last_fullsized":"2024-04-15T15:16:26.885719+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.885492+0000","last_change":"2024-04-15T15:16:25.889861+0000","last_active":"2024-04-15T15:16:26.885492+0000","last_peered":"2024-04-15T15:16:26.885492+0000","last_clean":"2024-04-15T15:16:26.885492+0000","last_became_active":"2024-04-15T15:16:25.889591+0000","last_became_peered":"2024-04-15T15:16:25.889591+0000","last_unstale":"2024-04-15T15:16:26.885492+0000","last_undegraded":"2024-04-15T15:16:26.885492+0000","last_fullsized":"2024-04-15T15:16:26.885492+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:25.819468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.884867+0000","last_change":"2024-04-15T15:16:24.891429+0000","last_active":"2024-04-15T15:16:26.884867+0000","last_peered":"2024-04-15T15:16:26.884867+0000","last_clean":"2024-04-15T15:16:26.884867+0000","last_became_active":"2024-04-15T15:16:24.891111+0000","last_became_peered":"2024-04-15T15:16:24.891111+0000","last_unstale":"2024-04-15T15:16:26.884867+0000","last_undegraded":"2024-04-15T15:16:26.884867+0000","last_fullsized":"2024-04-15T15:16:26.884867+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T18:53:18.563223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185587+0000","last_change":"2024-04-15T15:16:25.892154+0000","last_active":"2024-04-15T15:16:27.185587+0000","last_peered":"2024-04-15T15:16:27.185587+0000","last_clean":"2024-04-15T15:16:27.185587+0000","last_became_active":"2024-04-15T15:16:25.891901+0000","last_became_peered":"2024-04-15T15:16:25.891901+0000","last_unstale":"2024-04-15T15:16:27.185587+0000","last_undegraded":"2024-04-15T15:16:27.185587+0000","last_fullsized":"2024-04-15T15:16:27.185587+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T16:48:40.576675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:26.883631+0000","last_change":"2024-04-15T15:16:24.887452+0000","last_active":"2024-04-15T15:16:26.883631+0000","last_peered":"2024-04-15T15:16:26.883631+0000","last_clean":"2024-04-15T15:16:26.883631+0000","last_became_active":"2024-04-15T15:16:24.887136+0000","last_became_peered":"2024-04-15T15:16:24.887136+0000","last_unstale":"2024-04-15T15:16:26.883631+0000","last_undegraded":"2024-04-15T15:16:26.883631+0000","last_fullsized":"2024-04-15T15:16:26.883631+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T19:48:24.817169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-04-15T15:16:25.906655+0000","last_change":"2024-04-15T15:16:25.901054+0000","last_active":"2024-04-15T15:16:25.906655+0000","last_peered":"2024-04-15T15:16:25.906655+0000","last_clean":"2024-04-15T15:16:25.906655+0000","last_became_active":"2024-04-15T15:16:25.900784+0000","last_became_peered":"2024-04-15T15:16:25.900784+0000","last_unstale":"2024-04-15T15:16:25.906655+0000","last_undegraded":"2024-04-15T15:16:25.906655+0000","last_fullsized":"2024-04-15T15:16:25.906655+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:24.865017+0000","last_clean_scrub_stamp":"2024-04-15T15:16:24.865017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:32:58.561999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185529+0000","last_change":"2024-04-15T15:16:24.891549+0000","last_active":"2024-04-15T15:16:27.185529+0000","last_peered":"2024-04-15T15:16:27.185529+0000","last_clean":"2024-04-15T15:16:27.185529+0000","last_became_active":"2024-04-15T15:16:24.891268+0000","last_became_peered":"2024-04-15T15:16:24.891268+0000","last_unstale":"2024-04-15T15:16:27.185529+0000","last_undegraded":"2024-04-15T15:16:27.185529+0000","last_fullsized":"2024-04-15T15:16:27.185529+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:16:23.862718+0000","last_clean_scrub_stamp":"2024-04-15T15:16:23.862718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T22:42:59.916522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":203,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-15T15:16:27.185354+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:16:27.185354+0000","last_peered":"2024-04-15T15:16:27.185354+0000","last_clean":"2024-04-15T15:16:27.185354+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:16:27.185354+0000","last_undegraded":"2024-04-15T15:16:27.185354+0000","last_fullsized":"2024-04-15T15:16:27.185354+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542166,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1060864,"data_stored":786469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27452,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344388,"statfs":{"total":96636764160,"available":96608653312,"internally_reserved":0,"allocated":442368,"data_stored":188309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":187839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":446464,"data_stored":188331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":454656,"data_stored":189373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31612,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340228,"statfs":{"total":96636764160,"available":96604393472,"internally_reserved":0,"allocated":442368,"data_stored":187817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339644,"statfs":{"total":96636764160,"available":96603795456,"internally_reserved":0,"allocated":1036288,"data_stored":778129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:16:33.630 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-15T15:16:33.631 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-15T15:16:33.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.771+0000 7fcfd2ce4640 1 Processor -- start 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.771+0000 7fcfd2ce4640 1 -- start start 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd2ce4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcfcc0b3160 0x7fcfcc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd2ce4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcfcc0ae280 0x7fcfcc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd2ce4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfcc15cb50 con 0x7fcfcc0d0690 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd2ce4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc15bfd0 con 0x7fcfcc0ae280 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd2ce4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcfcc15c2b0 con 0x7fcfcc0b3160 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcfcc0ae280 0x7fcfcc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fcfcc0ae280 0x7fcfcc0ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47356/0 (socket says 172.21.15.70:47356) 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 -- 172.21.15.70:0/3396438331 learned_addr learned my addr 172.21.15.70:0/3396438331 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfcbfff640 1 --2- 172.21.15.70:0/3396438331 >> v2:172.21.15.96:3300/0 conn(0x7fcfcc0b3160 0x7fcfcc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 -- 172.21.15.70:0/3396438331 >> v2:172.21.15.96:3300/0 conn(0x7fcfcc0b3160 msgr2=0x7fcfcc15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 --2- 172.21.15.70:0/3396438331 >> v2:172.21.15.96:3300/0 conn(0x7fcfcc0b3160 0x7fcfcc15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.772+0000 7fcfd1281640 1 -- 172.21.15.70:0/3396438331 >> v1:172.21.15.96:6789/0 conn(0x7fcfcc0d0690 legacy=0x7fcfcc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:33.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.773+0000 7fcfd1281640 1 -- 172.21.15.70:0/3396438331 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc15c590 con 0x7fcfcc0ae280 2024-04-15T15:16:33.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.773+0000 7fcfd1281640 1 --2- 172.21.15.70:0/3396438331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc0ae710 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06fec0 tx=0x7fcfc402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad5deeae2e4cc751 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.773+0000 7fcfcb7fe640 1 -- 172.21.15.70:0/3396438331 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc402cb20 con 0x7fcfcc0ae280 2024-04-15T15:16:33.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.773+0000 7fcfcb7fe640 1 -- 172.21.15.70:0/3396438331 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc402ccc0 con 0x7fcfcc0ae280 2024-04-15T15:16:33.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.773+0000 7fcfcb7fe640 1 -- 172.21.15.70:0/3396438331 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4034610 con 0x7fcfcc0ae280 2024-04-15T15:16:33.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3396438331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0ae280 msgr2=0x7fcfcc0ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3396438331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc0ae710 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc06fec0 tx=0x7fcfc402bf80 comp rx=0 tx=0).stop 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3396438331 shutdown_connections 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3396438331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc0ae710 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3396438331 >> v2:172.21.15.96:3300/0 conn(0x7fcfcc0b3160 0x7fcfcc15b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3396438331 >> 172.21.15.70:0/3396438331 conn(0x7fcfcc06be00 msgr2=0x7fcfcc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3396438331 shutdown_connections 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.774+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3396438331 wait complete. 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.775+0000 7fcfd2ce4640 1 Processor -- start 2024-04-15T15:16:33.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.775+0000 7fcfd2ce4640 1 -- start start 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc15ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 0x7fcfcc15f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc15bbd0 con 0x7fcfcc0d0690 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcfcc0b4fb0 con 0x7fcfcc0ae280 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd2ce4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc15bd50 con 0x7fcfcc0b3160 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47372/0 (socket says 172.21.15.70:47372) 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 -- 172.21.15.70:0/3934923196 learned_addr learned my addr 172.21.15.70:0/3934923196 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd0a80640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc15ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfcbfff640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 0x7fcfcc15f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 -- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 msgr2=0x7fcfcc15f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 0x7fcfcc15f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 -- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc0ae280 msgr2=0x7fcfcc15ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc15ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfd1281640 1 -- 172.21.15.70:0/3934923196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc15d3b0 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.776+0000 7fcfcbfff640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 0x7fcfcc15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfd1281640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4036040 tx=0x7fcfc4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4043020 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfc403e450 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfcc15d650 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4009510 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.777+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfcc1f27e0 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.778+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfcc0689e0 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.778+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fcfc4046020 con 0x7fcfcc0d0690 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.779+0000 7fcfc97fa640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 0x7fcfb806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:33.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.779+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcfc40ae470 con 0x7fcfcc0d0690 2024-04-15T15:16:33.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.779+0000 7fcfd0a80640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 0x7fcfb806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:33.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.779+0000 7fcfd0a80640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 0x7fcfb806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc1f5bb0 tx=0x7fcfbc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:33.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:33.783+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfc404d030 con 0x7fcfcc0d0690 2024-04-15T15:16:34.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.030+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fcfcc0739e0 con 0x7fcfcc0d0690 2024-04-15T15:16:34.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.033+0000 7fcfc97fa640 1 -- 172.21.15.70:0/3934923196 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fcfc407c710 con 0x7fcfcc0d0690 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBGRB1mV0TQEhAAVAq/vgY21Iz2b2lmIPnQPQ== 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stdout: caps mds = "allow" 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stdout: caps mon = "allow r" 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-15T15:16:34.035 INFO:teuthology.orchestra.run.smithi070.stderr:updated caps for client.0 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 msgr2=0x7fcfb806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 0x7fcfb806edb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc1f5bb0 tx=0x7fcfbc009040 comp rx=0 tx=0).stop 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 msgr2=0x7fcfcc1f2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fcfc4036040 tx=0x7fcfc4004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 shutdown_connections 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfb806c8f0 0x7fcfb806edb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc0d0690 0x7fcfcc1f2fe0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc0b3160 0x7fcfcc15f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 --2- 172.21.15.70:0/3934923196 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc0ae280 0x7fcfcc15ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 >> 172.21.15.70:0/3934923196 conn(0x7fcfcc06be00 msgr2=0x7fcfcc06e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 shutdown_connections 2024-04-15T15:16:34.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.035+0000 7fcfd2ce4640 1 -- 172.21.15.70:0/3934923196 wait complete. 2024-04-15T15:16:34.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:34.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.151+0000 7f8b4e7ba640 1 Processor -- start 2024-04-15T15:16:34.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.151+0000 7f8b4e7ba640 1 -- start start 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4e7ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b480cd000 0x7f8b480d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4e7ba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 0x7f8b4809afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4e7ba640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b4815dbd0 con 0x7f8b481602d0 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4e7ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8b480cd460 con 0x7f8b480cd000 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4e7ba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8b480cd6a0 con 0x7f8b4809ab40 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b477fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8b480cd000 0x7f8b480d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b47fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8b481602d0 0x7f8b481606b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44400/0 (socket says 172.21.15.70:44400) 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b47fff640 1 -- 172.21.15.70:0/2881805946 learned_addr learned my addr 172.21.15.70:0/2881805946 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4cd57640 1 --2- 172.21.15.70:0/2881805946 >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 0x7f8b4809afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b477fe640 1 -- 172.21.15.70:0/2881805946 >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 msgr2=0x7f8b4809afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b477fe640 1 --2- 172.21.15.70:0/2881805946 >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 0x7f8b4809afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b477fe640 1 -- 172.21.15.70:0/2881805946 >> v1:172.21.15.70:6789/0 conn(0x7f8b481602d0 legacy=0x7f8b481606b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b477fe640 1 -- 172.21.15.70:0/2881805946 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b480cd980 con 0x7f8b480cd000 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b4cd57640 1 --2- 172.21.15.70:0/2881805946 >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 0x7f8b4809afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.152+0000 7f8b46ffd640 1 -- 172.21.15.70:0/2881805946 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 544654630 0 0) 0x7f8b4815dbd0 con 0x7f8b481602d0 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b477fe640 1 --2- 172.21.15.70:0/2881805946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b480cd000 0x7f8b480d0ef0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8b480d0470 tx=0x7f8b3c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db310a6cf4863bd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b46ffd640 1 -- 172.21.15.70:0/2881805946 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c035020 con 0x7f8b480cd000 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b46ffd640 1 -- 172.21.15.70:0/2881805946 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b3c031830 con 0x7f8b480cd000 2024-04-15T15:16:34.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b46ffd640 1 -- 172.21.15.70:0/2881805946 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c031c00 con 0x7f8b480cd000 2024-04-15T15:16:34.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/2881805946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b480cd000 msgr2=0x7f8b480d0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.153+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/2881805946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b480cd000 0x7f8b480d0ef0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8b480d0470 tx=0x7f8b3c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/2881805946 shutdown_connections 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/2881805946 >> v2:172.21.15.96:3300/0 conn(0x7f8b4809ab40 0x7f8b4809afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/2881805946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b480cd000 0x7f8b480d0ef0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/2881805946 >> 172.21.15.70:0/2881805946 conn(0x7f8b480d2460 msgr2=0x7f8b480d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/2881805946 shutdown_connections 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.154+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/2881805946 wait complete. 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.155+0000 7f8b4e7ba640 1 Processor -- start 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.155+0000 7f8b4e7ba640 1 -- start start 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b4809ab40 0x7f8b4815f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b480cd000 0x7f8b48156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480ce220 con 0x7f8b481602d0 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b480cdf40 con 0x7f8b480cd000 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8b480d1430 con 0x7f8b4809ab40 2024-04-15T15:16:34.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47398/0 (socket says 172.21.15.70:47398) 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 -- 172.21.15.70:0/1760253442 learned_addr learned my addr 172.21.15.70:0/1760253442 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 -- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b4809ab40 msgr2=0x7f8b4815f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b4809ab40 0x7f8b4815f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 -- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b480cd000 msgr2=0x7f8b48156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b480cd000 0x7f8b48156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 -- 172.21.15.70:0/1760253442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b4815e170 con 0x7f8b481602d0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4cd57640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000cd50 tx=0x7f8b4000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000f680 con 0x7f8b481602d0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b4000f840 con 0x7f8b481602d0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b4815e450 con 0x7f8b481602d0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b4000fc30 con 0x7f8b481602d0 2024-04-15T15:16:34.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.156+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b4815ea10 con 0x7f8b481602d0 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.157+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b480689e0 con 0x7f8b481602d0 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.158+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f8b400213f0 con 0x7f8b481602d0 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.158+0000 7f8b44ff9640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 0x7f8b3406ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.158+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8b4008ca40 con 0x7f8b481602d0 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.158+0000 7f8b47fff640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 0x7f8b3406ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.159+0000 7f8b47fff640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 0x7f8b3406ede0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b38007fd0 tx=0x7f8b38007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.163+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b40014030 con 0x7f8b481602d0 2024-04-15T15:16:34.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.373+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b480b4e20 con 0x7f8b481602d0 2024-04-15T15:16:34.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.373+0000 7f8b44ff9640 1 -- 172.21.15.70:0/1760253442 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7f8b4008ce20 con 0x7f8b481602d0 2024-04-15T15:16:34.375 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:34.375 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:16:27.226372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/2334246723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2334246723},{"type":"v1","addr":"172.21.15.96:6835","nonce":2334246723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4907,"qdb_cluster":[4907]},"id":2}]} 2024-04-15T15:16:34.376 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 15 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 msgr2=0x7f8b3406ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 0x7f8b3406ede0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b38007fd0 tx=0x7f8b38007920 comp rx=0 tx=0).stop 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 msgr2=0x7f8b480cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000cd50 tx=0x7f8b4000a650 comp rx=0 tx=0).stop 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 shutdown_connections 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b3406c920 0x7f8b3406ede0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.377 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b481602d0 0x7f8b480cb340 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b480cd000 0x7f8b48156630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 --2- 172.21.15.70:0/1760253442 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b4809ab40 0x7f8b4815f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 >> 172.21.15.70:0/1760253442 conn(0x7f8b480d2460 msgr2=0x7f8b480d04b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 shutdown_connections 2024-04-15T15:16:34.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.375+0000 7f8b4e7ba640 1 -- 172.21.15.70:0/1760253442 wait complete. 2024-04-15T15:16:34.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:25.228602+0000', 'modified': '2024-04-15T15:16:27.226372+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4907}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4907': {'gid': 4907, 'name': 'd', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.96:6835/2334246723', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 2334246723}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 2334246723}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4907, 'qdb_cluster': [4907]} 2024-04-15T15:16:34.791 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:16:34.791 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-15T15:16:34.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.895+0000 7fbf5e738640 1 Processor -- start 2024-04-15T15:16:34.897 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.895+0000 7fbf5e738640 1 -- start start 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5e738640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbf5815ec10 0x7fbf580b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5e738640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf58067e30 con 0x7fbf580b53c0 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5e738640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf580b3c10 con 0x7fbf5815e830 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5e738640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbf580b3ef0 con 0x7fbf5815ec10 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5ccd5640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbf580b53c0 0x7fbf580aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44410/0 (socket says 172.21.15.70:44410) 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf577fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbf5815ec10 0x7fbf580b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/3541938737 learned_addr learned my addr 172.21.15.70:0/3541938737 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:34.898 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf577fe640 1 -- 172.21.15.70:0/3541938737 >> v1:172.21.15.96:6789/0 conn(0x7fbf5815e830 legacy=0x7fbf58067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf577fe640 1 -- 172.21.15.70:0/3541938737 >> v1:172.21.15.70:6789/0 conn(0x7fbf580b53c0 legacy=0x7fbf580aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf56ffd640 1 -- 172.21.15.70:0/3541938737 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524651671 0 0) 0x7fbf58067e30 con 0x7fbf580b53c0 2024-04-15T15:16:34.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.896+0000 7fbf577fe640 1 -- 172.21.15.70:0/3541938737 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf580b41d0 con 0x7fbf5815ec10 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.897+0000 7fbf577fe640 1 --2- 172.21.15.70:0/3541938737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf5815ec10 0x7fbf580b4cb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0097a0 tx=0x7fbf4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f58ae7570ef17e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.897+0000 7fbf56ffd640 1 -- 172.21.15.70:0/3541938737 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c035020 con 0x7fbf5815ec10 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.897+0000 7fbf56ffd640 1 -- 172.21.15.70:0/3541938737 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf4c031830 con 0x7fbf5815ec10 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.897+0000 7fbf56ffd640 1 -- 172.21.15.70:0/3541938737 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c031c00 con 0x7fbf5815ec10 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 -- 172.21.15.70:0/3541938737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf5815ec10 msgr2=0x7fbf580b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.900 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 --2- 172.21.15.70:0/3541938737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf5815ec10 0x7fbf580b4cb0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c0097a0 tx=0x7fbf4c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 -- 172.21.15.70:0/3541938737 shutdown_connections 2024-04-15T15:16:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 --2- 172.21.15.70:0/3541938737 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf5815ec10 0x7fbf580b4cb0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 -- 172.21.15.70:0/3541938737 >> 172.21.15.70:0/3541938737 conn(0x7fbf58073360 msgr2=0x7fbf5806b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 -- 172.21.15.70:0/3541938737 shutdown_connections 2024-04-15T15:16:34.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.898+0000 7fbf5e738640 1 -- 172.21.15.70:0/3541938737 wait complete. 2024-04-15T15:16:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.899+0000 7fbf5e738640 1 Processor -- start 2024-04-15T15:16:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.899+0000 7fbf5e738640 1 -- start start 2024-04-15T15:16:34.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbf5815e830 0x7fbf5815ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5815d3b0 0x7fbf581e77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf5815ff60 con 0x7fbf580b53c0 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf580b4790 con 0x7fbf5815d3b0 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbf581600e0 con 0x7fbf5815e830 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47416/0 (socket says 172.21.15.70:47416) 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 -- 172.21.15.70:0/1070929043 learned_addr learned my addr 172.21.15.70:0/1070929043 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5815d3b0 0x7fbf581e77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf577fe640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbf5815e830 0x7fbf5815ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 -- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbf5815e830 msgr2=0x7fbf5815ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbf5815e830 0x7fbf5815ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 -- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5815d3b0 msgr2=0x7fbf581e77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5815d3b0 0x7fbf581e77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 -- 172.21.15.70:0/1070929043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf581ea390 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf57fff640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c480 tx=0x7fbf4800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf54ff9640 1 -- 172.21.15.70:0/1070929043 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf480052d0 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf54ff9640 1 -- 172.21.15.70:0/1070929043 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf48005470 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf581e9090 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf54ff9640 1 -- 172.21.15.70:0/1070929043 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48012870 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.900+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf581e87f0 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.902+0000 7fbf54ff9640 1 -- 172.21.15.70:0/1070929043 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fbf48012a10 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.902+0000 7fbf54ff9640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 0x7fbf4406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.902+0000 7fbf54ff9640 1 -- 172.21.15.70:0/1070929043 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbf4808b930 con 0x7fbf580b53c0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.902+0000 7fbf577fe640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 0x7fbf4406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.903+0000 7fbf577fe640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 0x7fbf4406edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c036000 tx=0x7fbf4c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.903+0000 7fbf613f5740 1 Processor -- start 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.903+0000 7fbf613f5740 1 -- start start 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651ce7cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce4d2f0 0x55651ce7d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55651cdb79c0 con 0x55651ce79e10 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55651cd5e650 con 0x55651ce4d2f0 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x55651cdb3980 con 0x55651ce7a1f0 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47430/0 (socket says 172.21.15.70:47430) 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 -- 172.21.15.70:0/2284459720 learned_addr learned my addr 172.21.15.70:0/2284459720 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf577fe640 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651ce7cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce4d2f0 0x55651ce7d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 -- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 msgr2=0x55651ce7cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651ce7cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 -- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce4d2f0 msgr2=0x55651ce7d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce4d2f0 0x55651ce7d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 -- 172.21.15.70:0/2284459720 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55651caca2d0 con 0x55651ce79e10 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000f30 tx=0x7fbf48058fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=960ca68b58e3ec48 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.904+0000 7fbf3dffb640 1 -- 172.21.15.70:0/2284459720 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48088070 con 0x55651ce79e10 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf3dffb640 1 -- 172.21.15.70:0/2284459720 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf48062140 con 0x55651ce79e10 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf3dffb640 1 -- 172.21.15.70:0/2284459720 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf48012d00 con 0x55651ce79e10 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 -- 172.21.15.70:0/2284459720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 msgr2=0x55651ce4cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbf48000f30 tx=0x7fbf48058fc0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 -- 172.21.15.70:0/2284459720 shutdown_connections 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce4d2f0 0x55651ce7d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651ce7cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2284459720 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce79e10 0x55651ce4cdb0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 -- 172.21.15.70:0/2284459720 >> 172.21.15.70:0/2284459720 conn(0x55651ce4f020 msgr2=0x55651ce97ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 -- 172.21.15.70:0/2284459720 shutdown_connections 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.905+0000 7fbf613f5740 1 -- 172.21.15.70:0/2284459720 wait complete. 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.906+0000 7fbf613f5740 1 -- start start 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.906+0000 7fbf613f5740 1 Processor -- start 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ceac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce921d0 0x55651ceab6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55651ce515c0 con 0x55651ce7a1f0 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55651cd71a10 con 0x55651ce79e10 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x55651ce509b0 con 0x55651ce921d0 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 10 client.0 _ll_register_callbacks cb 0x55651cdad250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47434/0 (socket says 172.21.15.70:47434) 2024-04-15T15:16:34.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 -- 172.21.15.70:0/512268182 learned_addr learned my addr 172.21.15.70:0/512268182 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce921d0 0x55651ceab6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf57fff640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ceac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 -- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce921d0 msgr2=0x55651ceab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce921d0 0x55651ceab6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 -- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 msgr2=0x55651ceac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ceac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 -- 172.21.15.70:0/512268182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55651cb946b0 con 0x55651ce7a1f0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf577fe640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c02bee0 tx=0x7fbf4c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2f0acc4c233902 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c055060 con 0x55651ce7a1f0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbf4c045440 con 0x55651ce7a1f0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf48062140 con 0x55651ce7a1f0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fbf4c05b050 con 0x55651ce7a1f0 2024-04-15T15:16:34.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.907+0000 7fbf613f5740 10 client.5730 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55651ce91700 con 0x55651ce7a1f0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf48062140 con 0x55651ce7a1f0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fbf4c05e020 con 0x55651ce7a1f0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 10 client.5730 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 10 client.5730 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 20 client.5730 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ce7f9a0 0x55651ceb6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 4 client.5730 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55651cdc53e0 con 0x55651ce7f9a0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf57fff640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ce7f9a0 0x55651ceb6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf36ffd640 10 client.5730 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.908+0000 7fbf57fff640 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ce7f9a0 0x55651ceb6340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebecb32fd2212003 server_cookie=50369e82a2e7f219 in_seq=0 out_seq=0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf36ffd640 1 -- 172.21.15.70:0/512268182 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55651cdc53e0 con 0x55651ce7f9a0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf36ffd640 10 client.5730 handle_command_reply: tid=0 2024-04-15T15:16:34.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 client.5730 shutdown 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 msgr2=0x55651cea7ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c02bee0 tx=0x7fbf4c02e000 comp rx=0 tx=0).stop 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 shutdown_connections 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ce7f9a0 0x55651ceb6340 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce921d0 0x55651ceab6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce7a1f0 0x55651cea7ae0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf36ffd640 0 client.5730 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 --2- 172.21.15.70:0/512268182 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ceac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 >> 172.21.15.70:0/512268182 conn(0x55651ce4d2f0 msgr2=0x55651ce9b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 shutdown_connections 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.909+0000 7fbf613f5740 1 -- 172.21.15.70:0/512268182 wait complete. 2024-04-15T15:16:34.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.910+0000 7fbf613f5740 20 client.5730 trim_cache size 0 max 16384 2024-04-15T15:16:34.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.927+0000 7fbf613f5740 1 Processor -- start 2024-04-15T15:16:34.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 -- start start 2024-04-15T15:16:34.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce50b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cebb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.931 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbf58149270 con 0x55651ce921d0 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbf581e5e60 con 0x55651ce79e10 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbf58148c20 con 0x55651ce7a1f0 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf5ccd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf5ccd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47438/0 (socket says 172.21.15.70:47438) 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.928+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/379550539 learned_addr learned my addr 172.21.15.70:0/379550539 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 msgr2=0x55651cebb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cebb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 msgr2=0x55651ce50b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce50b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/379550539 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55651ceaba10 con 0x55651ce921d0 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbf500075c0 tx=0x7fbf50007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa883b0aa19b854d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf3d7fa640 1 -- 172.21.15.70:0/379550539 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50014c90 con 0x55651ce921d0 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf3d7fa640 1 -- 172.21.15.70:0/379550539 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf50014e30 con 0x55651ce921d0 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf3d7fa640 1 -- 172.21.15.70:0/379550539 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50016620 con 0x55651ce921d0 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 -- 172.21.15.70:0/379550539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 msgr2=0x55651cebbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fbf500075c0 tx=0x7fbf50007a80 comp rx=0 tx=0).stop 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 -- 172.21.15.70:0/379550539 shutdown_connections 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cebbae0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cebb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 --2- 172.21.15.70:0/379550539 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce50b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 -- 172.21.15.70:0/379550539 >> 172.21.15.70:0/379550539 conn(0x55651ce4d2f0 msgr2=0x55651ce805e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.929+0000 7fbf613f5740 1 -- 172.21.15.70:0/379550539 shutdown_connections 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.930+0000 7fbf613f5740 1 -- 172.21.15.70:0/379550539 wait complete. 2024-04-15T15:16:34.934 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.930+0000 7fbf613f5740 1 -- start start 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.930+0000 7fbf613f5740 1 Processor -- start 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce7c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cea6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55651ce7f820 con 0x55651ce921d0 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x55651ce801c0 con 0x55651ce79e10 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x55651ce7fe70 con 0x55651ce7a1f0 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf613f5740 10 client.0 _ll_register_callbacks cb 0x7fbf4c031950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47454/0 (socket says 172.21.15.70:47454) 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/2144767052 learned_addr learned my addr 172.21.15.70:0/2144767052 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce7c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf577fe640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cea6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 msgr2=0x55651cea6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cea6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 msgr2=0x55651ce7c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce7c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf5ccd5640 1 -- 172.21.15.70:0/2144767052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55651ce78640 con 0x55651ce921d0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.931+0000 7fbf577fe640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cea6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf5ccd5640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbf5001a000 tx=0x7fbf50007ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63f8696703bfdb90 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf50016de0 con 0x55651ce921d0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbf50027440 con 0x55651ce921d0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf50014e30 con 0x55651ce921d0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fbf5002e050 con 0x55651ce921d0 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 10 client.5736 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:34.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x55651ce541e0 con 0x55651ce921d0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf50014e30 con 0x55651ce921d0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fbf50031020 con 0x55651ce921d0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 10 client.5736 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 10 client.5736 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 20 client.5736 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ceaa130 0x55651ceaa510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 4 client.5736 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "status"}) -- 0x55651cdbf250 con 0x55651ceaa130 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.932+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ceaa130 0x55651ceaa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf36ffd640 10 client.5736 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf57fff640 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ceaa130 0x55651ceaa510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2270fa5744c388c6 server_cookie=36a867fc79e4ffda in_seq=0 out_seq=0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf36ffd640 1 -- 172.21.15.70:0/2144767052 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x55651cdbf250 con 0x55651ceaa130 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf36ffd640 10 client.5736 handle_command_reply: tid=0 2024-04-15T15:16:34.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 client.5736 shutdown 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 msgr2=0x55651cf12090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fbf5001a000 tx=0x7fbf50007ec0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 shutdown_connections 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55651ce921d0 0x55651cf12090 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x55651ceaa130 0x55651ceaa510 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x55651ce7a1f0 0x55651cea6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 --2- 172.21.15.70:0/2144767052 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x55651ce79e10 0x55651ce7c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf36ffd640 0 client.5736 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 >> 172.21.15.70:0/2144767052 conn(0x55651ce4d2f0 msgr2=0x55651ce4bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 shutdown_connections 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 1 -- 172.21.15.70:0/2144767052 wait complete. 2024-04-15T15:16:34.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.933+0000 7fbf613f5740 20 client.5736 trim_cache size 0 max 16384 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "074678c8-a3ad-4fb8-89f1-2c7d457fbbe5", 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 4907, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 9.062881462, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 15, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 39, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 39, 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 16.194967372000001 2024-04-15T15:16:34.942 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 msgr2=0x7fbf4406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 0x7fbf4406edb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c036000 tx=0x7fbf4c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 msgr2=0x7fbf581e9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:34.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800c480 tx=0x7fbf4800c940 comp rx=0 tx=0).stop 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 shutdown_connections 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbf5815d3b0 0x7fbf581e77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbf5815e830 0x7fbf5815ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf4406c8f0 0x7fbf4406edb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 --2- 172.21.15.70:0/1070929043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbf580b53c0 0x7fbf581e9e50 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 >> 172.21.15.70:0/1070929043 conn(0x7fbf58073360 msgr2=0x7fbf580aea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 shutdown_connections 2024-04-15T15:16:34.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:34.934+0000 7fbf5e738640 1 -- 172.21.15.70:0/1070929043 wait complete. 2024-04-15T15:16:34.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:35.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.049+0000 7f40f70ba640 1 Processor -- start 2024-04-15T15:16:35.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.049+0000 7f40f70ba640 1 -- start start 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.049+0000 7f40f70ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f006f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.049+0000 7f40f70ba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40f006fbc0 0x7f40f00ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f70ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40f00aee30 con 0x7f40f00ae600 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f70ba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f40f0070000 con 0x7f40f006fbc0 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f70ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f40f00af110 con 0x7f40f0158de0 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f4e56640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f006f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40effff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f40f006fbc0 0x7f40f00ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f5657640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f40f00ae600 0x7f40f015efc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:35930/0 (socket says 172.21.15.70:35930) 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f5657640 1 -- 172.21.15.70:0/448899620 learned_addr learned my addr 172.21.15.70:0/448899620 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f4e56640 1 -- 172.21.15.70:0/448899620 >> v2:172.21.15.96:3300/0 conn(0x7f40f006fbc0 msgr2=0x7f40f00ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:35.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f4e56640 1 --2- 172.21.15.70:0/448899620 >> v2:172.21.15.96:3300/0 conn(0x7f40f006fbc0 0x7f40f00ae0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f4e56640 1 -- 172.21.15.70:0/448899620 >> v1:172.21.15.96:6789/0 conn(0x7f40f00ae600 legacy=0x7f40f015efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.050+0000 7f40f4e56640 1 -- 172.21.15.70:0/448899620 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f00af3f0 con 0x7f40f0158de0 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.051+0000 7f40f4e56640 1 --2- 172.21.15.70:0/448899620 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f006f680 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40e80097a0 tx=0x7f40e802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e83554166aaf75 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.051+0000 7f40ef7fe640 1 -- 172.21.15.70:0/448899620 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8035020 con 0x7f40f0158de0 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.051+0000 7f40ef7fe640 1 -- 172.21.15.70:0/448899620 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e8031830 con 0x7f40f0158de0 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.051+0000 7f40ef7fe640 1 -- 172.21.15.70:0/448899620 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8031c00 con 0x7f40f0158de0 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 -- 172.21.15.70:0/448899620 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 msgr2=0x7f40f006f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 --2- 172.21.15.70:0/448899620 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f006f680 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f40e80097a0 tx=0x7f40e802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 -- 172.21.15.70:0/448899620 shutdown_connections 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 --2- 172.21.15.70:0/448899620 >> v2:172.21.15.96:3300/0 conn(0x7f40f006fbc0 0x7f40f00ae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 --2- 172.21.15.70:0/448899620 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f006f680 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 -- 172.21.15.70:0/448899620 >> 172.21.15.70:0/448899620 conn(0x7f40f00b1f60 msgr2=0x7f40f00b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 -- 172.21.15.70:0/448899620 shutdown_connections 2024-04-15T15:16:35.055 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.052+0000 7f40f70ba640 1 -- 172.21.15.70:0/448899620 wait complete. 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.053+0000 7f40f70ba640 1 Processor -- start 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.053+0000 7f40f70ba640 1 -- start start 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f00ae600 0x7f40f0150290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f009d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f40f00aea70 con 0x7f40f006fbc0 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f40f00af9b0 con 0x7f40f00ae600 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f70ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f40f00aebf0 con 0x7f40f0158de0 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47470/0 (socket says 172.21.15.70:47470) 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 -- 172.21.15.70:0/2633178145 learned_addr learned my addr 172.21.15.70:0/2633178145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f5657640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f009d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40effff640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f00ae600 0x7f40f0150290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 -- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 msgr2=0x7f40f009d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f009d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 -- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f00ae600 msgr2=0x7f40f0150290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f00ae600 0x7f40f0150290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.054+0000 7f40f4e56640 1 -- 172.21.15.70:0/2633178145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f009daa0 con 0x7f40f006fbc0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40f4e56640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f40e802b910 tx=0x7f40e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8040070 con 0x7f40f006fbc0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40e803a930 con 0x7f40f006fbc0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f009c500 con 0x7f40f006fbc0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8035040 con 0x7f40f006fbc0 2024-04-15T15:16:35.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.055+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f009ca80 con 0x7f40f006fbc0 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.056+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40f00689e0 con 0x7f40f006fbc0 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.056+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f40e804b060 con 0x7f40f006fbc0 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.057+0000 7f40ed7fa640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 0x7f40d006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.057+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f40e80ad3e0 con 0x7f40f006fbc0 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.057+0000 7f40effff640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 0x7f40d006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:35.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.057+0000 7f40effff640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 0x7f40d006edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f40f00b09e0 tx=0x7f40e000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:35.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.061+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f40e8031800 con 0x7f40f006fbc0 2024-04-15T15:16:35.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.274+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f40f006e060 con 0x7f40f006fbc0 2024-04-15T15:16:35.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.274+0000 7f40ed7fa640 1 -- 172.21.15.70:0/2633178145 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f40e807b030 con 0x7f40f006fbc0 2024-04-15T15:16:35.276 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:35.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 msgr2=0x7f40d006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 0x7f40d006edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f40f00b09e0 tx=0x7f40e000b040 comp rx=0 tx=0).stop 2024-04-15T15:16:35.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 msgr2=0x7f40f015c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f40e802b910 tx=0x7f40e80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 shutdown_connections 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f40f0158de0 0x7f40f009d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f40f00ae600 0x7f40f0150290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f40d006c8f0 0x7f40d006edb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 --2- 172.21.15.70:0/2633178145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f40f006fbc0 0x7f40f015c290 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 >> 172.21.15.70:0/2633178145 conn(0x7f40f00b1f60 msgr2=0x7f40f00b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 shutdown_connections 2024-04-15T15:16:35.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35.276+0000 7f40f70ba640 1 -- 172.21.15.70:0/2633178145 wait complete. 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:35.289 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:35.290 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:35.290 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:35.290 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:35.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.290 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:16:35.306 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:35.306 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:35.306 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.307 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42656sec preferred_lft 42656sec 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c488:39ff:fe35:ded1/64 scope link 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 82:8a:03:3a:60:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::808a:3ff:fe3a:6003/64 scope link 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:35.308 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:35.309 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::6898:acff:fe89:2456/64 scope link 2024-04-15T15:16:35.309 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:35.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.309 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:35.374 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:16:35.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.375 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:16:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:16:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:35.433 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:35.434 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:16:35.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:35.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:35.573 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.573 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:16:35.589 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:35.589 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:35.589 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:35.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.590 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-15T15:16:35.658 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:16:35.658 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:35.659 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:35.715 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:35.715 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:35.715 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:35.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:35.839 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:35.896 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:35.897 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:35.898 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:35.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:35.899 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:35.952 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:35.952 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-15T15:16:35.995 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:36.023 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:36.082 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:36.083 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:36.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.084 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:36.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-15T15:16:36.082+0000 7ff1b3042480 -1 init, newargv = 0x563612cff280 newargc=15 2024-04-15T15:16:36.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[53692]: starting ceph client 2024-04-15T15:16:36.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[53692]: starting fuse 2024-04-15T15:16:36.100 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:16:36.100 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:36.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:36.100 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:16:36.100 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:36.101 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-15T15:16:36.102 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:16:36.222 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.53692.asok 2024-04-15T15:16:36.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:36.233 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53692.asok 2024-04-15T15:16:36.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.233 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53692.asok status 2024-04-15T15:16:36.371 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:36.371 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:16:36.371 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:36.371 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "53692", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/" 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5742, 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 5742 2024-04-15T15:16:36.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 72583039 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 72583039 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.5742 192.168.0.1:0/72583039", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/72583039", 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 1, 2024-04-15T15:16:36.373 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 15, 2024-04-15T15:16:36.374 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 39, 2024-04-15T15:16:36.374 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:36.374 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:16:36.374 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:16:36.374 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:36.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.380 DEBUG:teuthology.orchestra.run.smithi070:> stat --file-system '--printf=%T 2024-04-15T15:16:36.380 DEBUG:teuthology.orchestra.run.smithi070:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:36.396 INFO:teuthology.orchestra.run.smithi070.stdout:fuseblk 2024-04-15T15:16:36.396 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:36.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:36.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:36.608 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 Processor -- start 2024-04-15T15:16:36.609 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 -- start start 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25280b4200 con 0x7f2528155330 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f25280b52a0 con 0x7f2528072bb0 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.607+0000 7f252ddc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f252815c090 con 0x7f252815bc30 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2527fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2527fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:54982/0 (socket says 172.21.15.70:54982) 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2527fff640 1 -- 172.21.15.70:0/315285816 learned_addr learned my addr 172.21.15.70:0/315285816 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f25277fe640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f2528155330 0x7f25280b39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:35942/0 (socket says 172.21.15.70:35942) 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:51522/0 (socket says 172.21.15.70:51522) 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 -- 172.21.15.70:0/315285816 >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 msgr2=0x7f252806e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 --2- 172.21.15.70:0/315285816 >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 -- 172.21.15.70:0/315285816 >> v1:172.21.15.96:6789/0 conn(0x7f2528155330 legacy=0x7f25280b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f25267fc640 1 -- 172.21.15.70:0/315285816 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4073912195 0 0) 0x7f25280b4200 con 0x7f2528155330 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2526ffd640 1 -- 172.21.15.70:0/315285816 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25280b44e0 con 0x7f252815bc30 2024-04-15T15:16:36.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.608+0000 7f2527fff640 1 --2- 172.21.15.70:0/315285816 >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.609+0000 7f2526ffd640 1 --2- 172.21.15.70:0/315285816 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f25100097a0 tx=0x7f251002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9079db721e8724b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.609+0000 7f25267fc640 1 -- 172.21.15.70:0/315285816 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510035020 con 0x7f252815bc30 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.609+0000 7f25267fc640 1 -- 172.21.15.70:0/315285816 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2510031830 con 0x7f252815bc30 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.609+0000 7f25267fc640 1 -- 172.21.15.70:0/315285816 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510031c00 con 0x7f252815bc30 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 -- 172.21.15.70:0/315285816 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 msgr2=0x7f2528072670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 --2- 172.21.15.70:0/315285816 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f25100097a0 tx=0x7f251002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 -- 172.21.15.70:0/315285816 shutdown_connections 2024-04-15T15:16:36.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 --2- 172.21.15.70:0/315285816 >> v2:172.21.15.96:3300/0 conn(0x7f2528072bb0 0x7f252806e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 --2- 172.21.15.70:0/315285816 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f252815bc30 0x7f2528072670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 -- 172.21.15.70:0/315285816 >> 172.21.15.70:0/315285816 conn(0x7f25280b1f60 msgr2=0x7f25280b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 -- 172.21.15.70:0/315285816 shutdown_connections 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.610+0000 7f252ddc8640 1 -- 172.21.15.70:0/315285816 wait complete. 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.611+0000 7f252ddc8640 1 Processor -- start 2024-04-15T15:16:36.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.611+0000 7f252ddc8640 1 -- start start 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2528072bb0 0x7f25281517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2528155330 0x7f25280ad570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f25280b4aa0 con 0x7f252815bc30 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2528072fe0 con 0x7f2528155330 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f252ddc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f25280b4d80 con 0x7f2528072bb0 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47478/0 (socket says 172.21.15.70:47478) 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 -- 172.21.15.70:0/2728475519 learned_addr learned my addr 172.21.15.70:0/2728475519 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2526ffd640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2528155330 0x7f25280ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f25277fe640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2528072bb0 0x7f25281517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 -- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2528072bb0 msgr2=0x7f25281517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2528072bb0 0x7f25281517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 -- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2528155330 msgr2=0x7f25280ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2528155330 0x7f25280ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 -- 172.21.15.70:0/2728475519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25280adff0 con 0x7f252815bc30 2024-04-15T15:16:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.612+0000 7f2527fff640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2520010d40 tx=0x7f252000e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.620 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.613+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520004220 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.613+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25200043c0 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.613+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25280ae2d0 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.613+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520003830 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.613+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25280ae890 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.614+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25280b3390 con 0x7f252815bc30 2024-04-15T15:16:36.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.614+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2520020020 con 0x7f252815bc30 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.615+0000 7f251ffff640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 0x7f250806edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.615+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f252008e5c0 con 0x7f252815bc30 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.615+0000 7f25277fe640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 0x7f250806edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.615+0000 7f25277fe640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 0x7f250806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f25180012e0 tx=0x7f2518001630 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:36.626 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.619+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f25280b3390 con 0x7f252815bc30 2024-04-15T15:16:36.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.833+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2528154eb0 con 0x7f252815bc30 2024-04-15T15:16:36.835 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.834+0000 7f251ffff640 1 -- 172.21.15.70:0/2728475519 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f2528154eb0 con 0x7f252815bc30 2024-04-15T15:16:36.835 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:36.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 msgr2=0x7f250806edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 0x7f250806edb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f25180012e0 tx=0x7f2518001630 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 msgr2=0x7f2528157da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f2520010d40 tx=0x7f252000e730 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 shutdown_connections 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f250806c8f0 0x7f250806edb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f252815bc30 0x7f2528157da0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2528155330 0x7f25280ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 --2- 172.21.15.70:0/2728475519 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2528072bb0 0x7f25281517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.835+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 >> 172.21.15.70:0/2728475519 conn(0x7f25280b1f60 msgr2=0x7f252815c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.836+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 shutdown_connections 2024-04-15T15:16:36.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:36.836+0000 7f252ddc8640 1 -- 172.21.15.70:0/2728475519 wait complete. 2024-04-15T15:16:36.850 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:36.850 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:36.851 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:36.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.851 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-15T15:16:36.871 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-15T15:16:36.872 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42629sec preferred_lft 42629sec 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:36.873 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::c27:e4ff:fe37:c79c/64 scope link 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether f6:ed:c4:10:c9:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::f4ed:c4ff:fe10:c9d9/64 scope link 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::d096:56ff:fe49:5376/64 scope link 2024-04-15T15:16:36.874 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:36.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.875 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:36.944 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:16:36.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:36.946 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-15T15:16:37.005 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-15T15:16:37.005 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:37.005 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:37.005 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:37.005 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:37.005 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:37.005 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:16:37.005 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:37.006 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:37.006 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:37.006 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:16:37.099 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:37.144 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:37.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.151 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:16:37.167 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:37.167 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:37.167 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:37.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.167 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-04-15T15:16:37.236 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:16:37.236 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:37.236 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:37.291 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:37.292 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:37.292 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:37.347 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:37.417 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:37.474 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:37.475 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:37.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.476 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:37.530 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:37.530 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-15T15:16:37.574 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:37.603 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:37.660 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:37.661 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:37.662 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:37.663 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:37.664 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:37.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.665 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:37.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:2024-04-15T15:16:37.660+0000 7f82f8a49480 -1 init, newargv = 0x5624253eb090 newargc=15 2024-04-15T15:16:37.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45162]: starting ceph client 2024-04-15T15:16:37.666 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45162]: starting fuse 2024-04-15T15:16:37.681 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:16:37.681 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:16:37.681 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:37.681 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:37.682 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.1")) 2024-04-15T15:16:37.683 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-15T15:16:37.803 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.1.45162.asok 2024-04-15T15:16:37.806 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:37.813 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45162.asok 2024-04-15T15:16:37.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.814 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45162.asok status 2024-04-15T15:16:37.951 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "1", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "45162", 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/" 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5332, 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 5332 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3540247838 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 3540247838 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.5332 192.168.0.1:0/3540247838", 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/3540247838", 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 1, 2024-04-15T15:16:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 15, 2024-04-15T15:16:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 39, 2024-04-15T15:16:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-15T15:16:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-15T15:16:37.955 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-15T15:16:37.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.960 DEBUG:teuthology.orchestra.run.smithi096:> stat --file-system '--printf=%T 2024-04-15T15:16:37.960 DEBUG:teuthology.orchestra.run.smithi096:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:37.976 INFO:teuthology.orchestra.run.smithi096.stdout:fuseblk 2024-04-15T15:16:37.976 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:37.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:37.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:38.045 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds debug_mds 25 2024-04-15T15:16:38.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.163+0000 7f1a23bd4640 1 Processor -- start 2024-04-15T15:16:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.163+0000 7f1a23bd4640 1 -- start start 2024-04-15T15:16:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a23bd4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1a1c0d0690 0x7f1a1c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a23bd4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a23bd4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a1c0d0fb0 con 0x7f1a1c0b3160 2024-04-15T15:16:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a23bd4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1a1c15bda0 con 0x7f1a1c0d0690 2024-04-15T15:16:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a23bd4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c15cb50 con 0x7f1a1c0ae280 2024-04-15T15:16:38.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a22171640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a22171640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:51530/0 (socket says 172.21.15.70:51530) 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a22171640 1 -- 172.21.15.70:0/773622089 learned_addr learned my addr 172.21.15.70:0/773622089 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.164+0000 7f1a21970640 1 --2- 172.21.15.70:0/773622089 >> v2:172.21.15.96:3300/0 conn(0x7f1a1c0d0690 0x7f1a1c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a22171640 1 -- 172.21.15.70:0/773622089 >> v2:172.21.15.96:3300/0 conn(0x7f1a1c0d0690 msgr2=0x7f1a1c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a22171640 1 --2- 172.21.15.70:0/773622089 >> v2:172.21.15.96:3300/0 conn(0x7f1a1c0d0690 0x7f1a1c0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a2096e640 1 -- 172.21.15.70:0/773622089 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2317592524 0 0) 0x7f1a1c0d0fb0 con 0x7f1a1c0b3160 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a22171640 1 -- 172.21.15.70:0/773622089 >> v1:172.21.15.96:6789/0 conn(0x7f1a1c0b3160 legacy=0x7f1a1c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a22171640 1 -- 172.21.15.70:0/773622089 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c15bfd0 con 0x7f1a1c0ae280 2024-04-15T15:16:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.165+0000 7f1a22171640 1 --2- 172.21.15.70:0/773622089 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009e50 tx=0x7f1a1402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646e74d78db9bc0e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a2096e640 1 -- 172.21.15.70:0/773622089 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a1402cb00 con 0x7f1a1c0ae280 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a2096e640 1 -- 172.21.15.70:0/773622089 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a1402cca0 con 0x7f1a1c0ae280 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a2096e640 1 -- 172.21.15.70:0/773622089 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14034610 con 0x7f1a1c0ae280 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 -- 172.21.15.70:0/773622089 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 msgr2=0x7f1a1c0ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/773622089 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009e50 tx=0x7f1a1402bf60 comp rx=0 tx=0).stop 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 -- 172.21.15.70:0/773622089 shutdown_connections 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/773622089 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0ae710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/773622089 >> v2:172.21.15.96:3300/0 conn(0x7f1a1c0d0690 0x7f1a1c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 -- 172.21.15.70:0/773622089 >> 172.21.15.70:0/773622089 conn(0x7f1a1c06be00 msgr2=0x7f1a1c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 -- 172.21.15.70:0/773622089 shutdown_connections 2024-04-15T15:16:38.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.166+0000 7f1a23bd4640 1 -- 172.21.15.70:0/773622089 wait complete. 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.167+0000 7f1a23bd4640 1 Processor -- start 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.167+0000 7f1a23bd4640 1 -- start start 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0c8ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c0b3160 0x7f1a1c0c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c15c870 con 0x7f1a1c0d0690 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1c15c590 con 0x7f1a1c0b3160 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a23bd4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1a1c0b4fb0 con 0x7f1a1c0ae280 2024-04-15T15:16:38.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a22171640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a22171640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47494/0 (socket says 172.21.15.70:47494) 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a22171640 1 -- 172.21.15.70:0/510167306 learned_addr learned my addr 172.21.15.70:0/510167306 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a2116f640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c0b3160 0x7f1a1c0c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.168+0000 7f1a21970640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0c8ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 -- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 msgr2=0x7f1a1c0c8ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0c8ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 -- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c0b3160 msgr2=0x7f1a1c0c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c0b3160 0x7f1a1c0c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 -- 172.21.15.70:0/510167306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1c09c650 con 0x7f1a1c0d0690 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a22171640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009e50 tx=0x7f1a14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14043020 con 0x7f1a1c0d0690 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a1402ce50 con 0x7f1a1c0d0690 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a1c09c930 con 0x7f1a1c0d0690 2024-04-15T15:16:38.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a14009510 con 0x7f1a1c0d0690 2024-04-15T15:16:38.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.169+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a1c0c9000 con 0x7f1a1c0d0690 2024-04-15T15:16:38.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.170+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a1c073ca0 con 0x7f1a1c0d0690 2024-04-15T15:16:38.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.170+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f1a14046020 con 0x7f1a1c0d0690 2024-04-15T15:16:38.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.171+0000 7f1a127fc640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 0x7f1a0006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.171+0000 7f1a21970640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 0x7f1a0006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.171+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f1a140ae130 con 0x7f1a1c0d0690 2024-04-15T15:16:38.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.171+0000 7f1a21970640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 0x7f1a0006edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009b40 tx=0x7f1a18009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.175+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1a1c073ca0 con 0x7f1a1c0d0690 2024-04-15T15:16:38.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.350+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=debug_mds}] v 0) -- 0x7f1a1c0689e0 con 0x7f1a1c0d0690 2024-04-15T15:16:38.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.353+0000 7f1a127fc640 1 -- 172.21.15.70:0/510167306 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=debug_mds}]=0 v11)=0 v11) ==== 108+0+0 (secure 0 0 0) 0x7f1a1404d030 con 0x7f1a1c0d0690 2024-04-15T15:16:38.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 msgr2=0x7f1a0006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 0x7f1a0006edb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f1a18009b40 tx=0x7f1a18009190 comp rx=0 tx=0).stop 2024-04-15T15:16:38.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 msgr2=0x7f1a1c09d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f1a14009e50 tx=0x7f1a14004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 shutdown_connections 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1a0006c8f0 0x7f1a0006edb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1a1c0d0690 0x7f1a1c09d410 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1a1c0b3160 0x7f1a1c0c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 --2- 172.21.15.70:0/510167306 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1a1c0ae280 0x7f1a1c0c8ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 >> 172.21.15.70:0/510167306 conn(0x7f1a1c06be00 msgr2=0x7f1a1c0cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 shutdown_connections 2024-04-15T15:16:38.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.355+0000 7f1a23bd4640 1 -- 172.21.15.70:0/510167306 wait complete. 2024-04-15T15:16:38.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_cache_quiesce_splitauth true 2024-04-15T15:16:38.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.478+0000 7f2ee3ea5640 1 Processor -- start 2024-04-15T15:16:38.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.478+0000 7f2ee3ea5640 1 -- start start 2024-04-15T15:16:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee3ea5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc15fda0 0x7f2edc0ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee3ea5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc0cfc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee3ea5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2edc0d0190 con 0x7f2edc0ced30 2024-04-15T15:16:38.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee3ea5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2edc0b1f40 con 0x7f2edc15fda0 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee3ea5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2edc0cda30 con 0x7f2edc0cf800 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc15fda0 0x7f2edc0ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1440640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2edc0ced30 0x7f2edc0b1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44420/0 (socket says 172.21.15.70:44420) 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2edc15fda0 0x7f2edc0ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47506/0 (socket says 172.21.15.70:47506) 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1440640 1 -- 172.21.15.70:0/2765058633 learned_addr learned my addr 172.21.15.70:0/2765058633 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee2442640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc0cfc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2765058633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 msgr2=0x7f2edc0cfc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc0cfc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2765058633 >> v1:172.21.15.70:6789/0 conn(0x7f2edc0ced30 legacy=0x7f2edc0b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2765058633 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc0cdd10 con 0x7f2edc15fda0 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee0c3f640 1 -- 172.21.15.70:0/2765058633 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984356811 0 0) 0x7f2edc0d0190 con 0x7f2edc0ced30 2024-04-15T15:16:38.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.479+0000 7f2ee2442640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.480+0000 7f2ee1c41640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc15fda0 0x7f2edc0ce7f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80097a0 tx=0x7f2ed802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecf949d290d9d38f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.480+0000 7f2ee0c3f640 1 -- 172.21.15.70:0/2765058633 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8035020 con 0x7f2edc15fda0 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.480+0000 7f2ee0c3f640 1 -- 172.21.15.70:0/2765058633 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed8031830 con 0x7f2edc15fda0 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.480+0000 7f2ee0c3f640 1 -- 172.21.15.70:0/2765058633 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8031c00 con 0x7f2edc15fda0 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2765058633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc15fda0 msgr2=0x7f2edc0ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc15fda0 0x7f2edc0ce7f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2ed80097a0 tx=0x7f2ed802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2765058633 shutdown_connections 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc0cfc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2765058633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc15fda0 0x7f2edc0ce7f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2765058633 >> 172.21.15.70:0/2765058633 conn(0x7f2edc0b6870 msgr2=0x7f2edc0b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2765058633 shutdown_connections 2024-04-15T15:16:38.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.481+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2765058633 wait complete. 2024-04-15T15:16:38.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.482+0000 7f2ee3ea5640 1 Processor -- start 2024-04-15T15:16:38.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.482+0000 7f2ee3ea5640 1 -- start start 2024-04-15T15:16:38.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc09b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2edc15fda0 0x7f2edc06f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc160180 con 0x7f2edc0ced30 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2edc0ce2d0 con 0x7f2edc15fda0 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2edc0b0790 con 0x7f2edc0cf800 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47520/0 (socket says 172.21.15.70:47520) 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2119321290 learned_addr learned my addr 172.21.15.70:0/2119321290 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1440640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc09b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee2442640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2edc15fda0 0x7f2edc06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 msgr2=0x7f2edc09b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc09b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2edc15fda0 msgr2=0x7f2edc06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2edc15fda0 0x7f2edc06f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.488 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 -- 172.21.15.70:0/2119321290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2edc143aa0 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee1c41640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8040070 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed8039a00 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.483+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2edc143d40 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.484+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8035040 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.484+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2edc144300 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.484+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2edc0b27f0 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.485+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f2ed804b060 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.485+0000 7f2ed27fc640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 0x7f2ec006edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.486+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2ed80ad4e0 con 0x7f2edc0ced30 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.486+0000 7f2ee1440640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 0x7f2ec006edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.486+0000 7f2ee1440640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 0x7f2ec006edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2edc151b40 tx=0x7f2ecc007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.490+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2edc0b27f0 con 0x7f2edc0ced30 2024-04-15T15:16:38.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.666+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mds_cache_quiesce_splitauth}] v 0) -- 0x7f2edc0689e0 con 0x7f2edc0ced30 2024-04-15T15:16:38.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.669+0000 7f2ed27fc640 1 -- 172.21.15.70:0/2119321290 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_cache_quiesce_splitauth}]=0 v12)=0 v12) ==== 128+0+0 (secure 0 0 0) 0x7f2ed807b130 con 0x7f2edc0ced30 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 msgr2=0x7f2ec006edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 0x7f2ec006edb0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2edc151b40 tx=0x7f2ecc007d50 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 msgr2=0x7f2edc1f1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8000c00 tx=0x7f2ed80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 shutdown_connections 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2edc15fda0 0x7f2edc06f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2edc0cf800 0x7f2edc09b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2ec006c8f0 0x7f2ec006edb0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 --2- 172.21.15.70:0/2119321290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2edc0ced30 0x7f2edc1f1920 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 >> 172.21.15.70:0/2119321290 conn(0x7f2edc0b6870 msgr2=0x7f2edc15eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 shutdown_connections 2024-04-15T15:16:38.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.671+0000 7f2ee3ea5640 1 -- 172.21.15.70:0/2119321290 wait complete. 2024-04-15T15:16:38.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs subvol_quiesce --mode=777 2024-04-15T15:16:38.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.789+0000 7f420e567640 1 Processor -- start 2024-04-15T15:16:38.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.789+0000 7f420e567640 1 -- start start 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.789+0000 7f420e567640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f420815eae0 0x7f42080b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f420e567640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f420806c0f0 0x7f42080ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f420e567640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42080b2310 con 0x7f420809a9d0 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f420e567640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f42080b6850 con 0x7f420815eae0 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f420e567640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f42080b6600 con 0x7f420806c0f0 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f420815eae0 0x7f42080b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f420815eae0 0x7f42080b60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47536/0 (socket says 172.21.15.70:47536) 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f42077fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f420809a9d0 0x7f420806b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44426/0 (socket says 172.21.15.70:44426) 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 -- 172.21.15.70:0/874145434 learned_addr learned my addr 172.21.15.70:0/874145434 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:38.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f420cb04640 1 --2- 172.21.15.70:0/874145434 >> v2:172.21.15.96:3300/0 conn(0x7f420806c0f0 0x7f42080ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 -- 172.21.15.70:0/874145434 >> v2:172.21.15.96:3300/0 conn(0x7f420806c0f0 msgr2=0x7f42080ae350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:38.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 --2- 172.21.15.70:0/874145434 >> v2:172.21.15.96:3300/0 conn(0x7f420806c0f0 0x7f42080ae350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 -- 172.21.15.70:0/874145434 >> v1:172.21.15.70:6789/0 conn(0x7f420809a9d0 legacy=0x7f420806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.790+0000 7f4207fff640 1 -- 172.21.15.70:0/874145434 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42080b25b0 con 0x7f420815eae0 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f4207fff640 1 --2- 172.21.15.70:0/874145434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420815eae0 0x7f42080b60c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f42080b6780 tx=0x7f41f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270df513ee72bf24 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f4206ffd640 1 -- 172.21.15.70:0/874145434 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8035020 con 0x7f420815eae0 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f4206ffd640 1 -- 172.21.15.70:0/874145434 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41f8031870 con 0x7f420815eae0 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f4206ffd640 1 -- 172.21.15.70:0/874145434 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41f8031c40 con 0x7f420815eae0 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f420e567640 1 -- 172.21.15.70:0/874145434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420815eae0 msgr2=0x7f42080b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.791+0000 7f420e567640 1 --2- 172.21.15.70:0/874145434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420815eae0 0x7f42080b60c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f42080b6780 tx=0x7f41f802b910 comp rx=0 tx=0).stop 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 -- 172.21.15.70:0/874145434 shutdown_connections 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 --2- 172.21.15.70:0/874145434 >> v2:172.21.15.96:3300/0 conn(0x7f420806c0f0 0x7f42080ae350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 --2- 172.21.15.70:0/874145434 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420815eae0 0x7f42080b60c0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 -- 172.21.15.70:0/874145434 >> 172.21.15.70:0/874145434 conn(0x7f42080b6a20 msgr2=0x7f42080b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:38.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 -- 172.21.15.70:0/874145434 shutdown_connections 2024-04-15T15:16:38.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.792+0000 7f420e567640 1 -- 172.21.15.70:0/874145434 wait complete. 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.793+0000 7f420e567640 1 Processor -- start 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.793+0000 7f420e567640 1 -- start start 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f420806c0f0 0x7f4208145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 0x7f42080af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42080b2b70 con 0x7f420809a9d0 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f42080b1d70 con 0x7f420815eae0 2024-04-15T15:16:38.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f420815eec0 con 0x7f420806c0f0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47550/0 (socket says 172.21.15.70:47550) 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 -- 172.21.15.70:0/3751297313 learned_addr learned my addr 172.21.15.70:0/3751297313 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f4207fff640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f420806c0f0 0x7f4208145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420cb04640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 0x7f42080af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 -- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f420806c0f0 msgr2=0x7f4208145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f420806c0f0 0x7f4208145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 -- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 msgr2=0x7f42080af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 0x7f42080af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 -- 172.21.15.70:0/3751297313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42081f4c90 con 0x7f420809a9d0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420cb04640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 0x7f42080af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f42077fe640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00c400 tx=0x7f41fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0074e0 con 0x7f420809a9d0 2024-04-15T15:16:38.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41fc007680 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc004a00 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42081f4f70 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.794+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420809db90 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.795+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42080689e0 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.796+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f41fc01a020 con 0x7f420809a9d0 2024-04-15T15:16:38.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.796+0000 7f4204ff9640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 0x7f41f406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:38.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.796+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f41fc08b520 con 0x7f420809a9d0 2024-04-15T15:16:38.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.796+0000 7f4207fff640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 0x7f41f406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:38.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.797+0000 7f4207fff640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 0x7f41f406edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f41f8036000 tx=0x7f41f803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:38.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:38.800+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f41fc059740 con 0x7f420809a9d0 2024-04-15T15:16:39.002 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.000+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "mode": "777", "target": ["mon-mgr", ""]}) -- 0x7f4208070090 con 0x7f41f406c8f0 2024-04-15T15:16:39.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:16:39.004+0000 7f41b824d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:39.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:16:39.004+0000 7f41b824d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:39.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:16:39.004+0000 7f41b824d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:39.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:16:39.004+0000 7f41b824d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:39.006 INFO:tasks.ceph.mgr.y.smithi070.stderr:2024-04-15T15:16:39.004+0000 7f41b824d640 -1 client.0 error registering admin socket command: (17) File exists 2024-04-15T15:16:39.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.033+0000 7f4204ff9640 1 -- 172.21.15.70:0/3751297313 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4208070090 con 0x7f41f406c8f0 2024-04-15T15:16:39.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 msgr2=0x7f41f406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 0x7f41f406edb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f41f8036000 tx=0x7f41f803c040 comp rx=0 tx=0).stop 2024-04-15T15:16:39.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 msgr2=0x7f42080af2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00c400 tx=0x7f41fc00c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 shutdown_connections 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f420815eae0 0x7f42080af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f41f406c8f0 0x7f41f406edb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f420809a9d0 0x7f42080af2a0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 --2- 172.21.15.70:0/3751297313 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f420806c0f0 0x7f4208145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 >> 172.21.15.70:0/3751297313 conn(0x7f42080b6a20 msgr2=0x7f420815d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.035+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 shutdown_connections 2024-04-15T15:16:39.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.036+0000 7f420e567640 1 -- 172.21.15.70:0/3751297313 wait complete. 2024-04-15T15:16:39.048 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs subvol_quiesce 2024-04-15T15:16:39.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.153+0000 7fd5f8806640 1 Processor -- start 2024-04-15T15:16:39.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.153+0000 7fd5f8806640 1 -- start start 2024-04-15T15:16:39.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f8806640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd5f4072d80 0x7fd5f4071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f8806640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f415db40 con 0x7fd5f40d0da0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f8806640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd5f415cfc0 con 0x7fd5f40af3e0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f8806640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd5f415d2a0 con 0x7fd5f4072d80 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f2d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd5f40d0da0 0x7fd5f406e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44438/0 (socket says 172.21.15.70:44438) 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1192035299 learned_addr learned my addr 172.21.15.70:0/1192035299 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.154+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1259794860 0 0) 0x7fd5f415db40 con 0x7fd5f40d0da0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5dc003520 con 0x7fd5f40d0da0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f259c640 1 --2- 172.21.15.70:0/1192035299 >> v2:172.21.15.96:3300/0 conn(0x7fd5f4072d80 0x7fd5f4071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1445136580 0 0) 0x7fd5dc003520 con 0x7fd5f40d0da0 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 >> v2:172.21.15.96:3300/0 conn(0x7fd5f4072d80 msgr2=0x7fd5f4071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 --2- 172.21.15.70:0/1192035299 >> v2:172.21.15.96:3300/0 conn(0x7fd5f4072d80 0x7fd5f4071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 >> v1:172.21.15.96:6789/0 conn(0x7fd5f40af3e0 legacy=0x7fd5f40d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f415d580 con 0x7fd5f40d0da0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fd5ec0019f0 con 0x7fd5f40d0da0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 >> v1:172.21.15.70:6789/0 conn(0x7fd5f40d0da0 legacy=0x7fd5f406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc0039b0 0x7fd5dc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 0x7fd5dc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc0051c0 0x7fd5dc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f415d2a0 con 0x7fd5dc004800 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5dc00c160 con 0x7fd5dc0039b0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd5dc00c360 con 0x7fd5dc0051c0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 0x7fd5dc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f259c640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc0039b0 0x7fd5dc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f2d9d640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc0051c0 0x7fd5dc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 -- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc0051c0 msgr2=0x7fd5dc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc0051c0 0x7fd5dc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 -- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc0039b0 msgr2=0x7fd5dc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5dc0039b0 0x7fd5dc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f1d9b640 1 -- 172.21.15.70:0/1192035299 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5dc003520 con 0x7fd5dc004800 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f2d9d640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc0051c0 0x7fd5dc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.155+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1192035299 reap_dead start 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.156+0000 7fd5f1d9b640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 0x7fd5dc004c80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8001750 tx=0x7fd5dc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a293c2b39b5ad657 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.156+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8001370 con 0x7fd5dc004800 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.156+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5e80052d0 con 0x7fd5dc004800 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.156+0000 7fd5f159a640 1 -- 172.21.15.70:0/1192035299 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8003830 con 0x7fd5dc004800 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 -- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 msgr2=0x7fd5dc004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 0x7fd5dc004c80 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fd5e8001750 tx=0x7fd5dc005830 comp rx=0 tx=0).stop 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 -- 172.21.15.70:0/1192035299 shutdown_connections 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1192035299 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5dc004800 0x7fd5dc004c80 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 -- 172.21.15.70:0/1192035299 >> 172.21.15.70:0/1192035299 conn(0x7fd5f406be00 msgr2=0x7fd5f406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 -- 172.21.15.70:0/1192035299 shutdown_connections 2024-04-15T15:16:39.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.157+0000 7fd5f8806640 1 -- 172.21.15.70:0/1192035299 wait complete. 2024-04-15T15:16:39.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.158+0000 7fd5f8806640 1 Processor -- start 2024-04-15T15:16:39.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.158+0000 7fd5f8806640 1 -- start start 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc004800 0x7fd5f40b0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5f41534f0 0x7fd5f4153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f415cd10 con 0x7fd5f41470e0 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f415cb90 con 0x7fd5f41534f0 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f8806640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd5f40af840 con 0x7fd5dc004800 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47596/0 (socket says 172.21.15.70:47596) 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1363900873 learned_addr learned my addr 172.21.15.70:0/1363900873 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f1d9b640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5f41534f0 0x7fd5f4153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f259c640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc004800 0x7fd5f40b0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc004800 msgr2=0x7fd5f40b0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc004800 0x7fd5f40b0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5f41534f0 msgr2=0x7fd5f4153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5f41534f0 0x7fd5f4153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 -- 172.21.15.70:0/1363900873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f4157250 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.159+0000 7fd5f2d9d640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec002040 tx=0x7fd5ec00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.160+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec00eb90 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.160+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd5ec018070 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.160+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f4147aa0 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.160+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5ec013420 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.160+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f41577d0 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.161+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f40689e0 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.161+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fd5ec0135c0 con 0x7fd5f41470e0 2024-04-15T15:16:39.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.161+0000 7fd5e2ffd640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 0x7fd5d406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.162+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fd5ec08bf60 con 0x7fd5f41470e0 2024-04-15T15:16:39.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.162+0000 7fd5f259c640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 0x7fd5d406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.162+0000 7fd5f259c640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 0x7fd5d406edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009c40 tx=0x7fd5e4009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.166+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd5ec05a200 con 0x7fd5f41470e0 2024-04-15T15:16:39.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.369+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "subvol_quiesce", "target": ["mon-mgr", ""]}) -- 0x7fd5f40b2930 con 0x7fd5d406c8f0 2024-04-15T15:16:39.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.374+0000 7fd5e2ffd640 1 -- 172.21.15.70:0/1363900873 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+70 (secure 0 0 0) 0x7fd5f40b2930 con 0x7fd5d406c8f0 2024-04-15T15:16:39.375 INFO:teuthology.orchestra.run.smithi070.stdout:/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 msgr2=0x7fd5d406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 0x7fd5d406edb0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd5e4009c40 tx=0x7fd5e4009190 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 msgr2=0x7fd5f4147560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fd5ec002040 tx=0x7fd5ec00e530 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 shutdown_connections 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd5d406c8f0 0x7fd5d406edb0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd5f41470e0 0x7fd5f4147560 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd5f41534f0 0x7fd5f4153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 --2- 172.21.15.70:0/1363900873 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd5dc004800 0x7fd5f40b0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 >> 172.21.15.70:0/1363900873 conn(0x7fd5f406be00 msgr2=0x7fd5f415bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.378 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 shutdown_connections 2024-04-15T15:16:39.379 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.376+0000 7fd5f8806640 1 -- 172.21.15.70:0/1363900873 wait complete. 2024-04-15T15:16:39.389 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:39.389 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:39.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:39.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.547+0000 7f911efbd640 1 Processor -- start 2024-04-15T15:16:39.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.547+0000 7f911efbd640 1 -- start start 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f911efbd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9118072d80 0x7f9118071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f911efbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f91180af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f911efbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f911815db40 con 0x7f91180af3e0 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f911efbd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f911815cfc0 con 0x7f9118072d80 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f911efbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f911815d2a0 con 0x7f91180d0da0 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f9117fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f91180af3e0 0x7f91180d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44440/0 (socket says 172.21.15.70:44440) 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.548+0000 7f9117fff640 1 -- 172.21.15.70:0/3489707923 learned_addr learned my addr 172.21.15.70:0/3489707923 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 --2- 172.21.15.70:0/3489707923 >> v2:172.21.15.96:3300/0 conn(0x7f9118072d80 0x7f9118071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911d55a640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f91180af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1532488273 0 0) 0x7f911815db40 con 0x7f91180af3e0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9100003520 con 0x7f91180af3e0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1269240922 0 0) 0x7f9100003520 con 0x7f91180af3e0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 msgr2=0x7f91180af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f91180af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 >> v2:172.21.15.96:3300/0 conn(0x7f9118072d80 msgr2=0x7f9118071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 --2- 172.21.15.70:0/3489707923 >> v2:172.21.15.96:3300/0 conn(0x7f9118072d80 0x7f9118071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f911815d580 con 0x7f91180af3e0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f910c001670 con 0x7f91180af3e0 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 >> v1:172.21.15.70:6789/0 conn(0x7f91180af3e0 legacy=0x7f91180d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91000039b0 0x7f9100003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f9100004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9100004d40 0x7f91000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f911815cfc0 con 0x7f91000039b0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f911815d2a0 con 0x7f9100004d40 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f910000b9f0 con 0x7f91180d0da0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91000039b0 0x7f9100003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f9117fff640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f9100004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911d55a640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9100004d40 0x7f91000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 -- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 msgr2=0x7f9100004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91180d0da0 0x7f9100004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 -- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9100004d40 msgr2=0x7f91000051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9100004d40 0x7f91000051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.549+0000 7f911cd59640 1 -- 172.21.15.70:0/3489707923 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9100003520 con 0x7f91000039b0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.550+0000 7f911cd59640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91000039b0 0x7f9100003dd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9108009ac0 tx=0x7f9100005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db44643561240ba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.550+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108039070 con 0x7f91000039b0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.550+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f910802c710 con 0x7f91000039b0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.550+0000 7f91177fe640 1 -- 172.21.15.70:0/3489707923 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9108009690 con 0x7f91000039b0 2024-04-15T15:16:39.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 -- 172.21.15.70:0/3489707923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91000039b0 msgr2=0x7f9100003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 --2- 172.21.15.70:0/3489707923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91000039b0 0x7f9100003dd0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9108009ac0 tx=0x7f9100005830 comp rx=0 tx=0).stop 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911d55a640 1 -- 172.21.15.70:0/3489707923 reap_dead start 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 -- 172.21.15.70:0/3489707923 shutdown_connections 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 -- 172.21.15.70:0/3489707923 >> 172.21.15.70:0/3489707923 conn(0x7f911806be00 msgr2=0x7f911806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 -- 172.21.15.70:0/3489707923 shutdown_connections 2024-04-15T15:16:39.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.551+0000 7f911efbd640 1 -- 172.21.15.70:0/3489707923 wait complete. 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.552+0000 7f911efbd640 1 Processor -- start 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.552+0000 7f911efbd640 1 -- start start 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91000039b0 0x7f91180cbac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118157250 0x7f91181576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f911815cd10 con 0x7f91180b3360 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f911815cb90 con 0x7f9118157250 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911efbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f91180af840 con 0x7f91000039b0 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47616/0 (socket says 172.21.15.70:47616) 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 -- 172.21.15.70:0/29559169 learned_addr learned my addr 172.21.15.70:0/29559169 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:39.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f9117fff640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118157250 0x7f91181576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911cd59640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91000039b0 0x7f91180cbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 -- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91000039b0 msgr2=0x7f91180cbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91000039b0 0x7f91180cbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 -- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118157250 msgr2=0x7f91181576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118157250 0x7f91181576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 -- 172.21.15.70:0/29559169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91181f42f0 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.553+0000 7f911d55a640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f91180b54e0 tx=0x7f91100115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.554+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110011e40 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.554+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f911001b070 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.554+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9110003990 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.554+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91180b40e0 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.554+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91180b46a0 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.555+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91180689e0 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.555+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7f9110005d10 con 0x7f91180b3360 2024-04-15T15:16:39.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.555+0000 7f91157fa640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 0x7f910406edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.556+0000 7f911cd59640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 0x7f910406edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.556+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f911008bfb0 con 0x7f91180b3360 2024-04-15T15:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.556+0000 7f911cd59640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 0x7f910406edb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9108036000 tx=0x7f9108030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.560+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f911005a250 con 0x7f91180b3360 2024-04-15T15:16:39.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.734+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f91180cc640 con 0x7f91180b3360 2024-04-15T15:16:39.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.735+0000 7f91157fa640 1 -- 172.21.15.70:0/29559169 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16882 (secure 0 0 0) 0x7f9110059b50 con 0x7f91180b3360 2024-04-15T15:16:39.738 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:39.738 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:26.878225+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.737+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 msgr2=0x7f910406edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.737+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 0x7f910406edb0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f9108036000 tx=0x7f9108030000 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 msgr2=0x7f91180b37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f91180b54e0 tx=0x7f91100115a0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 shutdown_connections 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f910406c8f0 0x7f910406edb0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91180b3360 0x7f91180b37e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9118157250 0x7f91181576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 --2- 172.21.15.70:0/29559169 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f91000039b0 0x7f91180cbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 >> 172.21.15.70:0/29559169 conn(0x7f911806be00 msgr2=0x7f911815aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 shutdown_connections 2024-04-15T15:16:39.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.738+0000 7f911efbd640 1 -- 172.21.15.70:0/29559169 wait complete. 2024-04-15T15:16:39.751 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:39.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:39.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.856+0000 7fc030b50640 1 Processor -- start 2024-04-15T15:16:39.858 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.856+0000 7fc030b50640 1 -- start start 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.857+0000 7fc030b50640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.857+0000 7fc030b50640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 0x7fc02c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.857+0000 7fc030b50640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc02c0b47a0 con 0x7fc02c0d2360 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.857+0000 7fc030b50640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc02c0b53b0 con 0x7fc02c0d3270 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.857+0000 7fc030b50640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc02c0b4a80 con 0x7fc02c15e700 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02ad9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 0x7fc02c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:51644/0 (socket says 172.21.15.70:51644) 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02ad9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 0x7fc02c0cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:55140/0 (socket says 172.21.15.70:55140) 2024-04-15T15:16:39.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 -- 172.21.15.70:0/1093763245 learned_addr learned my addr 172.21.15.70:0/1093763245 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:39.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc029d9b640 1 --1- 172.21.15.70:0/1093763245 >> v1:172.21.15.96:6789/0 conn(0x7fc02c0d2360 0x7fc02c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:35984/0 (socket says 172.21.15.70:35984) 2024-04-15T15:16:39.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 -- 172.21.15.70:0/1093763245 >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 msgr2=0x7fc02c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:39.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 --2- 172.21.15.70:0/1093763245 >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 0x7fc02c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 -- 172.21.15.70:0/1093763245 >> v1:172.21.15.96:6789/0 conn(0x7fc02c0d2360 legacy=0x7fc02c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.858+0000 7fc02a59c640 1 -- 172.21.15.70:0/1093763245 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c0b4d60 con 0x7fc02c15e700 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.859+0000 7fc02a59c640 1 --2- 172.21.15.70:0/1093763245 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009ea0 tx=0x7fc01c02c030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dbabfc3281ddfc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.859+0000 7fc02959a640 1 -- 172.21.15.70:0/1093763245 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c02cbd0 con 0x7fc02c15e700 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.859+0000 7fc02959a640 1 -- 172.21.15.70:0/1093763245 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc01c02cd70 con 0x7fc02c15e700 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.859+0000 7fc02959a640 1 -- 172.21.15.70:0/1093763245 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c034610 con 0x7fc02c15e700 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 -- 172.21.15.70:0/1093763245 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 msgr2=0x7fc02c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 --2- 172.21.15.70:0/1093763245 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc01c009ea0 tx=0x7fc01c02c030 comp rx=0 tx=0).stop 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 -- 172.21.15.70:0/1093763245 shutdown_connections 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 --2- 172.21.15.70:0/1093763245 >> v2:172.21.15.96:3300/0 conn(0x7fc02c0d3270 0x7fc02c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 --2- 172.21.15.70:0/1093763245 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b5840 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 -- 172.21.15.70:0/1093763245 >> 172.21.15.70:0/1093763245 conn(0x7fc02c08bf60 msgr2=0x7fc02c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 -- 172.21.15.70:0/1093763245 shutdown_connections 2024-04-15T15:16:39.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.860+0000 7fc030b50640 1 -- 172.21.15.70:0/1093763245 wait complete. 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.861+0000 7fc030b50640 1 Processor -- start 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.861+0000 7fc030b50640 1 -- start start 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc02c0d3270 0x7fc02c0b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c0b5d80 con 0x7fc02c0d2360 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc02c0b4450 con 0x7fc02c0d3270 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc030b50640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc02c0d1d40 con 0x7fc02c15e700 2024-04-15T15:16:39.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47620/0 (socket says 172.21.15.70:47620) 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 -- 172.21.15.70:0/3504906824 learned_addr learned my addr 172.21.15.70:0/3504906824 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc029d9b640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc02c0d3270 0x7fc02c0b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02ad9d640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 -- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 msgr2=0x7fc02c0b9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 -- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc02c0d3270 msgr2=0x7fc02c0b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc02c0d3270 0x7fc02c0b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 -- 172.21.15.70:0/3504906824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc02c0cd480 con 0x7fc02c0d2360 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.862+0000 7fc02a59c640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc01c036040 tx=0x7fc01c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.863+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c043020 con 0x7fc02c0d2360 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.863+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc02c0cd760 con 0x7fc02c0d2360 2024-04-15T15:16:39.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.863+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc02c0cdd20 con 0x7fc02c0d2360 2024-04-15T15:16:39.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.863+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc01c03e450 con 0x7fc02c0d2360 2024-04-15T15:16:39.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.863+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc01c009510 con 0x7fc02c0d2360 2024-04-15T15:16:39.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.864+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc02c0689e0 con 0x7fc02c0d2360 2024-04-15T15:16:39.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.864+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fc01c046020 con 0x7fc02c0d2360 2024-04-15T15:16:39.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.865+0000 7fc01affd640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 0x7fc00c06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:39.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.865+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fc01c0adf90 con 0x7fc02c0d2360 2024-04-15T15:16:39.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.865+0000 7fc029d9b640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 0x7fc00c06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:39.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.865+0000 7fc029d9b640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 0x7fc00c06edb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0c7ee0 tx=0x7fc020007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:39.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:39.869+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc01c04d030 con 0x7fc02c0d2360 2024-04-15T15:16:40.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.045+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc02c0ba500 con 0x7fc02c0d2360 2024-04-15T15:16:40.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.045+0000 7fc01affd640 1 -- 172.21.15.70:0/3504906824 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7fc01c07bbf0 con 0x7fc02c0d2360 2024-04-15T15:16:40.047 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 msgr2=0x7fc00c06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 0x7fc00c06edb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0c7ee0 tx=0x7fc020007e80 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 msgr2=0x7fc02c0b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc01c036040 tx=0x7fc01c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 shutdown_connections 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc02c15e700 0x7fc02c0b9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc02c0d3270 0x7fc02c0b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc00c06c8f0 0x7fc00c06edb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 --2- 172.21.15.70:0/3504906824 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc02c0d2360 0x7fc02c0b73f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 >> 172.21.15.70:0/3504906824 conn(0x7fc02c08bf60 msgr2=0x7fc02c0d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 shutdown_connections 2024-04-15T15:16:40.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.047+0000 7fc030b50640 1 -- 172.21.15.70:0/3504906824 wait complete. 2024-04-15T15:16:40.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:40.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.165+0000 7fac054c8640 1 Processor -- start 2024-04-15T15:16:40.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.165+0000 7fac054c8640 1 -- start start 2024-04-15T15:16:40.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fac054c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 0x7fac000d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fac054c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fac0015eae0 0x7fac000d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fac054c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac000b47a0 con 0x7fac0015e700 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fac054c8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fac000b53b0 con 0x7fac0015eae0 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fac054c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fac000b4a80 con 0x7fac000d2360 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabff7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fac0015eae0 0x7fac000d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabfeffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fac0015e700 0x7fac000b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44442/0 (socket says 172.21.15.70:44442) 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabff7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fac0015eae0 0x7fac000d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47626/0 (socket says 172.21.15.70:47626) 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabfeffd640 1 -- 172.21.15.70:0/3140025408 learned_addr learned my addr 172.21.15.70:0/3140025408 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabff7fe640 1 -- 172.21.15.70:0/3140025408 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 msgr2=0x7fac000d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabfe7fc640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 0x7fac000d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabff7fe640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 0x7fac000d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabff7fe640 1 -- 172.21.15.70:0/3140025408 >> v1:172.21.15.70:6789/0 conn(0x7fac0015e700 legacy=0x7fac000b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.166+0000 7fabfdffb640 1 -- 172.21.15.70:0/3140025408 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 261852600 0 0) 0x7fac000b47a0 con 0x7fac0015e700 2024-04-15T15:16:40.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabff7fe640 1 -- 172.21.15.70:0/3140025408 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac000b4d60 con 0x7fac0015eae0 2024-04-15T15:16:40.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabfe7fc640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 0x7fac000d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabff7fe640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac000d30a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fabf80097a0 tx=0x7fabf802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be5e79fd5fc8494 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabfdffb640 1 -- 172.21.15.70:0/3140025408 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8035020 con 0x7fac0015eae0 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabfdffb640 1 -- 172.21.15.70:0/3140025408 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf8031830 con 0x7fac0015eae0 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.167+0000 7fabfdffb640 1 -- 172.21.15.70:0/3140025408 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8031c00 con 0x7fac0015eae0 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 -- 172.21.15.70:0/3140025408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 msgr2=0x7fac000d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac000d30a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fabf80097a0 tx=0x7fabf802b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 -- 172.21.15.70:0/3140025408 shutdown_connections 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac000d30a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 --2- 172.21.15.70:0/3140025408 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac000d2360 0x7fac000d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.171 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 -- 172.21.15.70:0/3140025408 >> 172.21.15.70:0/3140025408 conn(0x7fac0008bf60 msgr2=0x7fac000d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 -- 172.21.15.70:0/3140025408 shutdown_connections 2024-04-15T15:16:40.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.168+0000 7fac054c8640 1 -- 172.21.15.70:0/3140025408 wait complete. 2024-04-15T15:16:40.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.169+0000 7fac054c8640 1 Processor -- start 2024-04-15T15:16:40.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.169+0000 7fac054c8640 1 -- start start 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 0x7fac00152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac0015e700 0x7fac000b9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fac000d1d40 con 0x7fac0015eae0 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fac000b4450 con 0x7fac000d2360 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fac054c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fac000d1ec0 con 0x7fac0015e700 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47642/0 (socket says 172.21.15.70:47642) 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 -- 172.21.15.70:0/1815943739 learned_addr learned my addr 172.21.15.70:0/1815943739 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:40.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabfeffd640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 0x7fac00152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabfe7fc640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac0015e700 0x7fac000b9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 -- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac0015e700 msgr2=0x7fac000b9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac0015e700 0x7fac000b9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 -- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 msgr2=0x7fac00152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 0x7fac00152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabff7fe640 1 -- 172.21.15.70:0/1815943739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac000b7180 con 0x7fac0015eae0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.170+0000 7fabfeffd640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 0x7fac00152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fabff7fe640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fabf8034040 tx=0x7fabf80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8041070 con 0x7fac0015eae0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fabf8004230 con 0x7fac0015eae0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac000b7f00 con 0x7fac0015eae0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabf8035040 con 0x7fac0015eae0 2024-04-15T15:16:40.176 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.171+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac000b7440 con 0x7fac0015eae0 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.172+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac000ce460 con 0x7fac0015eae0 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.172+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 5) ==== 91040+0+0 (secure 0 0 0) 0x7fabf8038020 con 0x7fac0015eae0 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.173+0000 7fabe77fe640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 0x7fabec06edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.173+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fabf80ad700 con 0x7fac0015eae0 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.173+0000 7fabfeffd640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 0x7fabec06edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.173+0000 7fabfeffd640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 0x7fabec06edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fabf0001460 tx=0x7fabf0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.179 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.177+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fabf807b9a0 con 0x7fac0015eae0 2024-04-15T15:16:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.352+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fac000689e0 con 0x7fac0015eae0 2024-04-15T15:16:40.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.352+0000 7fabe77fe640 1 -- 172.21.15.70:0/1815943739 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7fabf807b2a0 con 0x7fac0015eae0 2024-04-15T15:16:40.354 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 msgr2=0x7fabec06edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 0x7fabec06edb0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fabf0001460 tx=0x7fabf0009040 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 msgr2=0x7fac00160ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fabf8034040 tx=0x7fabf80046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 shutdown_connections 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fabec06c8f0 0x7fabec06edb0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fac0015eae0 0x7fac00160ad0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fac0015e700 0x7fac000b9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 --2- 172.21.15.70:0/1815943739 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fac000d2360 0x7fac00152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 >> 172.21.15.70:0/1815943739 conn(0x7fac0008bf60 msgr2=0x7fac000cfb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 shutdown_connections 2024-04-15T15:16:40.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.355+0000 7fac054c8640 1 -- 172.21.15.70:0/1815943739 wait complete. 2024-04-15T15:16:40.371 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.475+0000 7fd6497aa640 1 Processor -- start 2024-04-15T15:16:40.477 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.475+0000 7fd6497aa640 1 -- start start 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6497aa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd6440d3270 0x7fd6440cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6497aa640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6440b47a0 con 0x7fd6440d2360 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6497aa640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6440b4a80 con 0x7fd64415e700 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6497aa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd6440b53b0 con 0x7fd6440d3270 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6437fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd6440d3270 0x7fd6440cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd63a7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd6440d2360 0x7fd6440d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44458/0 (socket says 172.21.15.70:44458) 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd63a7fc640 1 -- 172.21.15.70:0/2574103344 learned_addr learned my addr 172.21.15.70:0/2574103344 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6437fe640 1 -- 172.21.15.70:0/2574103344 >> v1:172.21.15.96:6789/0 conn(0x7fd64415e700 legacy=0x7fd6440b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6437fe640 1 -- 172.21.15.70:0/2574103344 >> v1:172.21.15.70:6789/0 conn(0x7fd6440d2360 legacy=0x7fd6440d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6427fc640 1 -- 172.21.15.70:0/2574103344 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3780109556 0 0) 0x7fd6440b47a0 con 0x7fd6440d2360 2024-04-15T15:16:40.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.476+0000 7fd6437fe640 1 -- 172.21.15.70:0/2574103344 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6440b4d60 con 0x7fd6440d3270 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6437fe640 1 --2- 172.21.15.70:0/2574103344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cf370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd6440b5530 tx=0x7fd63c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6bc1b20547351 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6427fc640 1 -- 172.21.15.70:0/2574103344 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c039070 con 0x7fd6440d3270 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6427fc640 1 -- 172.21.15.70:0/2574103344 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd63c02cc50 con 0x7fd6440d3270 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6427fc640 1 -- 172.21.15.70:0/2574103344 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd63c034440 con 0x7fd6440d3270 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6497aa640 1 -- 172.21.15.70:0/2574103344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 msgr2=0x7fd6440cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.477+0000 7fd6497aa640 1 --2- 172.21.15.70:0/2574103344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cf370 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd6440b5530 tx=0x7fd63c02bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.478+0000 7fd6497aa640 1 -- 172.21.15.70:0/2574103344 shutdown_connections 2024-04-15T15:16:40.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.478+0000 7fd6497aa640 1 --2- 172.21.15.70:0/2574103344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cf370 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.478+0000 7fd6497aa640 1 -- 172.21.15.70:0/2574103344 >> 172.21.15.70:0/2574103344 conn(0x7fd64408bf60 msgr2=0x7fd6440d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.478+0000 7fd6497aa640 1 -- 172.21.15.70:0/2574103344 shutdown_connections 2024-04-15T15:16:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.478+0000 7fd6497aa640 1 -- 172.21.15.70:0/2574103344 wait complete. 2024-04-15T15:16:40.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.479+0000 7fd6497aa640 1 Processor -- start 2024-04-15T15:16:40.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.479+0000 7fd6497aa640 1 -- start start 2024-04-15T15:16:40.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd6440d2360 0x7fd64415d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.483 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 0x7fd6440cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6440d1d40 con 0x7fd6440d3270 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd6440b4450 con 0x7fd64415e700 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd6440d1ec0 con 0x7fd6440d2360 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47664/0 (socket says 172.21.15.70:47664) 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 -- 172.21.15.70:0/3401638068 learned_addr learned my addr 172.21.15.70:0/3401638068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6437fe640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 0x7fd6440cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 -- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd6440d2360 msgr2=0x7fd64415d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd6440d2360 0x7fd64415d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 -- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 msgr2=0x7fd6440cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.484 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 0x7fd6440cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 -- 172.21.15.70:0/3401638068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64409f820 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6437fe640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 0x7fd6440cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63a7fc640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c500 tx=0x7fd62c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c018070 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd62c007580 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd62c004a20 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6440ce570 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.480+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64415ffa0 con 0x7fd6440d3270 2024-04-15T15:16:40.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.481+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6440689e0 con 0x7fd6440d3270 2024-04-15T15:16:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.482+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd62c01b070 con 0x7fd6440d3270 2024-04-15T15:16:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.482+0000 7fd63bfff640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 0x7fd62406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.482+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fd62c08afa0 con 0x7fd6440d3270 2024-04-15T15:16:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.483+0000 7fd642ffd640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 0x7fd62406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.483+0000 7fd642ffd640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 0x7fd62406ee40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd634009650 tx=0x7fd634006bb0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.487+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd62c0592e0 con 0x7fd6440d3270 2024-04-15T15:16:40.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.662+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6440ba500 con 0x7fd6440d3270 2024-04-15T15:16:40.664 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.662+0000 7fd63bfff640 1 -- 172.21.15.70:0/3401638068 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fd62c058be0 con 0x7fd6440d3270 2024-04-15T15:16:40.664 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 msgr2=0x7fd62406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 0x7fd62406ee40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd634009650 tx=0x7fd634006bb0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 msgr2=0x7fd6440cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c500 tx=0x7fd62c00c9c0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 shutdown_connections 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd64415e700 0x7fd6440cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd62406c980 0x7fd62406ee40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd6440d3270 0x7fd6440cd880 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 --2- 172.21.15.70:0/3401638068 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd6440d2360 0x7fd64415d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 >> 172.21.15.70:0/3401638068 conn(0x7fd64408bf60 msgr2=0x7fd64406cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 shutdown_connections 2024-04-15T15:16:40.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.665+0000 7fd6497aa640 1 -- 172.21.15.70:0/3401638068 wait complete. 2024-04-15T15:16:40.678 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:40.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.780+0000 7fb215c37640 1 Processor -- start 2024-04-15T15:16:40.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.780+0000 7fb215c37640 1 -- start start 2024-04-15T15:16:40.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.780+0000 7fb215c37640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100b3160 0x7fb21015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb215c37640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb2100ae280 0x7fb2100ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb215c37640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21015cb50 con 0x7fb2100d0690 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb215c37640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb21015bfd0 con 0x7fb2100ae280 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb215c37640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb21015c2b0 con 0x7fb2100b3160 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb2100ae280 0x7fb2100ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb2100d0690 0x7fb2100d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44464/0 (socket says 172.21.15.70:44464) 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20f7fe640 1 -- 172.21.15.70:0/683209081 learned_addr learned my addr 172.21.15.70:0/683209081 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20effd640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100b3160 0x7fb21015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20ffff640 1 -- 172.21.15.70:0/683209081 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100b3160 msgr2=0x7fb21015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20ffff640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100b3160 0x7fb21015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20ffff640 1 -- 172.21.15.70:0/683209081 >> v1:172.21.15.70:6789/0 conn(0x7fb2100d0690 legacy=0x7fb2100d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20e7fc640 1 -- 172.21.15.70:0/683209081 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1700462280 0 0) 0x7fb21015cb50 con 0x7fb2100d0690 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.781+0000 7fb20ffff640 1 -- 172.21.15.70:0/683209081 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21015c590 con 0x7fb2100ae280 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb20ffff640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100ae280 0x7fb2100ae6e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1baf617ebfbdc60 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb20e7fc640 1 -- 172.21.15.70:0/683209081 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208039070 con 0x7fb2100ae280 2024-04-15T15:16:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb20e7fc640 1 -- 172.21.15.70:0/683209081 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20802cc50 con 0x7fb2100ae280 2024-04-15T15:16:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb20e7fc640 1 -- 172.21.15.70:0/683209081 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208034440 con 0x7fb2100ae280 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 -- 172.21.15.70:0/683209081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100ae280 msgr2=0x7fb2100ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100ae280 0x7fb2100ae6e0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fb208009d90 tx=0x7fb20802bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 -- 172.21.15.70:0/683209081 shutdown_connections 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100ae280 0x7fb2100ae6e0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 --2- 172.21.15.70:0/683209081 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100b3160 0x7fb21015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 -- 172.21.15.70:0/683209081 >> 172.21.15.70:0/683209081 conn(0x7fb21006be00 msgr2=0x7fb21006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 -- 172.21.15.70:0/683209081 shutdown_connections 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.782+0000 7fb215c37640 1 -- 172.21.15.70:0/683209081 wait complete. 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.784+0000 7fb215c37640 1 Processor -- start 2024-04-15T15:16:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.784+0000 7fb215c37640 1 -- start start 2024-04-15T15:16:40.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.784+0000 7fb215c37640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100ae280 0x7fb21014e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 0x7fb2100cf510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.789 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb21015bbd0 con 0x7fb2100d0690 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb2100b4fb0 con 0x7fb2100b3160 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb21015bd50 con 0x7fb2100ae280 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47680/0 (socket says 172.21.15.70:47680) 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 -- 172.21.15.70:0/996258497 learned_addr learned my addr 172.21.15.70:0/996258497 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20effd640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 0x7fb2100cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20f7fe640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100ae280 0x7fb21014e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 -- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100ae280 msgr2=0x7fb21014e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100ae280 0x7fb21014e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 -- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 msgr2=0x7fb2100cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 0x7fb2100cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 -- 172.21.15.70:0/996258497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21015de40 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20effd640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 0x7fb2100cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb20ffff640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb20802c4b0 tx=0x7fb208004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb208039050 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb208045070 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21015e120 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20803e610 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.785+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21015e6e0 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.786+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2100689e0 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.787+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb20803e7b0 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.787+0000 7fb1f7fff640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 0x7fb1fc06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.787+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fb2080ae840 con 0x7fb2100d0690 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.787+0000 7fb20f7fe640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 0x7fb1fc06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:40.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.788+0000 7fb20f7fe640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 0x7fb1fc06ee70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb200000f60 tx=0x7fb200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:40.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.792+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb20807caf0 con 0x7fb2100d0690 2024-04-15T15:16:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.982+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb21015b690 con 0x7fb2100d0690 2024-04-15T15:16:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.982+0000 7fb1f7fff640 1 -- 172.21.15.70:0/996258497 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7fb20807c3f0 con 0x7fb2100d0690 2024-04-15T15:16:40.984 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 msgr2=0x7fb1fc06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 0x7fb1fc06ee70 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb200000f60 tx=0x7fb200009040 comp rx=0 tx=0).stop 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 msgr2=0x7fb210158690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb20802c4b0 tx=0x7fb208004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 shutdown_connections 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb1fc06c9b0 0x7fb1fc06ee70 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb2100d0690 0x7fb210158690 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb2100b3160 0x7fb2100cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 --2- 172.21.15.70:0/996258497 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb2100ae280 0x7fb21014e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:40.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 >> 172.21.15.70:0/996258497 conn(0x7fb21006be00 msgr2=0x7fb2100726b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:40.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 shutdown_connections 2024-04-15T15:16:40.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:40.984+0000 7fb215c37640 1 -- 172.21.15.70:0/996258497 wait complete. 2024-04-15T15:16:40.997 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:40.997 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:41.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.154+0000 7f787a504640 1 Processor -- start 2024-04-15T15:16:41.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.154+0000 7f787a504640 1 -- start start 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.155+0000 7f787a504640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 0x7f7874072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.155+0000 7f787a504640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7874072c60 0x7f787406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.155+0000 7f787a504640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78740b4190 con 0x7f7874158770 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.155+0000 7f787a504640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f78740b4da0 con 0x7f7874072c60 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.155+0000 7f787a504640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f78740b4470 con 0x7f78740b5230 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f786bfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 0x7f7874072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7878aa1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f7874072c60 0x7f787406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7873fff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f7874158770 0x7f787415bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:36010/0 (socket says 172.21.15.70:36010) 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f786bfff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 0x7f7874072720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:51694/0 (socket says 172.21.15.70:51694) 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7873fff640 1 -- 172.21.15.70:0/977205744 learned_addr learned my addr 172.21.15.70:0/977205744 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7878aa1640 1 -- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 msgr2=0x7f7874072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7878aa1640 1 --2- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 0x7f7874072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7878aa1640 1 -- 172.21.15.70:0/977205744 >> v1:172.21.15.96:6789/0 conn(0x7f7874158770 legacy=0x7f787415bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f7878aa1640 1 -- 172.21.15.70:0/977205744 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78740b4750 con 0x7f7874072c60 2024-04-15T15:16:41.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.156+0000 7f78737fe640 1 -- 172.21.15.70:0/977205744 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863227270 0 0) 0x7f78740b4190 con 0x7f7874158770 2024-04-15T15:16:41.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f7878aa1640 1 --2- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874072c60 0x7f787406dd30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f786c00aea0 tx=0x7f786c02cfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd21bee630749572 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f78737fe640 1 -- 172.21.15.70:0/977205744 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c02db40 con 0x7f7874072c60 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f78737fe640 1 -- 172.21.15.70:0/977205744 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f786c02dce0 con 0x7f7874072c60 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f78737fe640 1 -- 172.21.15.70:0/977205744 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f786c008660 con 0x7f7874072c60 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f787a504640 1 -- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874072c60 msgr2=0x7f787406dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f787a504640 1 --2- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874072c60 0x7f787406dd30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f786c00aea0 tx=0x7f786c02cfa0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f787a504640 1 -- 172.21.15.70:0/977205744 shutdown_connections 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.157+0000 7f787a504640 1 --2- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874072c60 0x7f787406dd30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.158+0000 7f787a504640 1 --2- 172.21.15.70:0/977205744 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78740b5230 0x7f7874072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.158+0000 7f787a504640 1 -- 172.21.15.70:0/977205744 >> 172.21.15.70:0/977205744 conn(0x7f787406be00 msgr2=0x7f787406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.158+0000 7f787a504640 1 -- 172.21.15.70:0/977205744 shutdown_connections 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.158+0000 7f787a504640 1 -- 172.21.15.70:0/977205744 wait complete. 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 Processor -- start 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 -- start start 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7874072c60 0x7f787414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874158770 0x7f7874155b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7874158b50 con 0x7f78740b5230 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f7874071fa0 con 0x7f7874158770 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f787a504640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f78740b3d20 con 0x7f7874072c60 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.159+0000 7f786bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47692/0 (socket says 172.21.15.70:47692) 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 -- 172.21.15.70:0/1148359406 learned_addr learned my addr 172.21.15.70:0/1148359406 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f7878aa1640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874158770 0x7f7874155b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f7873fff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7874072c60 0x7f787414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 -- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7874072c60 msgr2=0x7f787414d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7874072c60 0x7f787414d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 -- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874158770 msgr2=0x7f7874155b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874158770 0x7f7874155b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 -- 172.21.15.70:0/1148359406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78741520e0 con 0x7f78740b5230 2024-04-15T15:16:41.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f786bfff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f785c00e9b0 tx=0x7f785c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c017070 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c00c9a0 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78741523c0 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c00cd70 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.160+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7874152980 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.161+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78740689e0 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.162+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f785c01a070 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.162+0000 7f78717fa640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 0x7f785406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.162+0000 7f7873fff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 0x7f785406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.162+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f785c08bfd0 con 0x7f78740b5230 2024-04-15T15:16:41.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.163+0000 7f7873fff640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 0x7f785406ee40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7864007fd0 tx=0x7f7864007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.167+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f785c059c30 con 0x7f78740b5230 2024-04-15T15:16:41.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.345+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78740ad810 con 0x7f78740b5230 2024-04-15T15:16:41.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.347+0000 7f78717fa640 1 -- 172.21.15.70:0/1148359406 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16882 (secure 0 0 0) 0x7f785c05df30 con 0x7f78740b5230 2024-04-15T15:16:41.349 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:41.349 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:26.878225+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:41.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 msgr2=0x7f785406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 0x7f785406ee40 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7864007fd0 tx=0x7f7864007de0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 msgr2=0x7f7874153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f785c00e9b0 tx=0x7f785c00ee70 comp rx=0 tx=0).stop 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 shutdown_connections 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7874158770 0x7f7874155b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f785406c980 0x7f785406ee40 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f78740b5230 0x7f7874153180 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 --2- 172.21.15.70:0/1148359406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f7874072c60 0x7f787414d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 >> 172.21.15.70:0/1148359406 conn(0x7f787406be00 msgr2=0x7f787406dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 shutdown_connections 2024-04-15T15:16:41.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.349+0000 7f787a504640 1 -- 172.21.15.70:0/1148359406 wait complete. 2024-04-15T15:16:41.363 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:41.364 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.468+0000 7fcb9ff15640 1 Processor -- start 2024-04-15T15:16:41.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.468+0000 7fcb9ff15640 1 -- start start 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9ff15640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcb980d0690 0x7fcb980d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9ff15640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 0x7fcb9815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9ff15640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb980d0fb0 con 0x7fcb9815ef70 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9ff15640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcb9815cb50 con 0x7fcb980d0690 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9ff15640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcb9815bfd0 con 0x7fcb980b3160 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9e4b2640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcb9815ef70 0x7fcb9815f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44478/0 (socket says 172.21.15.70:44478) 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9e4b2640 1 -- 172.21.15.70:0/281340368 learned_addr learned my addr 172.21.15.70:0/281340368 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.469+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/281340368 >> v2:172.21.15.96:3300/0 conn(0x7fcb980d0690 0x7fcb980d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9d4b0640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 0x7fcb9815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9ccaf640 1 -- 172.21.15.70:0/281340368 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3407236558 0 0) 0x7fcb980d0fb0 con 0x7fcb9815ef70 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9ccaf640 1 -- 172.21.15.70:0/281340368 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcb84003520 con 0x7fcb9815ef70 2024-04-15T15:16:41.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 msgr2=0x7fcb9815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 0x7fcb9815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/281340368 >> v1:172.21.15.70:6789/0 conn(0x7fcb9815ef70 legacy=0x7fcb9815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/281340368 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9815c2b0 con 0x7fcb980d0690 2024-04-15T15:16:41.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.470+0000 7fcb9d4b0640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 0x7fcb9815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb980d0a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009d90 tx=0x7fcb9402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bc190ec731e3e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ccaf640 1 -- 172.21.15.70:0/281340368 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94039070 con 0x7fcb980d0690 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ccaf640 1 -- 172.21.15.70:0/281340368 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb9402cc50 con 0x7fcb980d0690 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ccaf640 1 -- 172.21.15.70:0/281340368 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94034440 con 0x7fcb980d0690 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 -- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 msgr2=0x7fcb980d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb980d0a70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009d90 tx=0x7fcb9402bea0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 -- 172.21.15.70:0/281340368 shutdown_connections 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb980b3160 0x7fcb9815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/281340368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb980d0a70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.471+0000 7fcb9ff15640 1 -- 172.21.15.70:0/281340368 >> 172.21.15.70:0/281340368 conn(0x7fcb9806be00 msgr2=0x7fcb9806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.472+0000 7fcb9ff15640 1 -- 172.21.15.70:0/281340368 shutdown_connections 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.472+0000 7fcb9ff15640 1 -- 172.21.15.70:0/281340368 wait complete. 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.473+0000 7fcb9ff15640 1 Processor -- start 2024-04-15T15:16:41.475 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.473+0000 7fcb9ff15640 1 -- start start 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.473+0000 7fcb9ff15640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb9809a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 0x7fcb9809a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcb980b4fb0 con 0x7fcb980b3160 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9815c870 con 0x7fcb980d0690 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcb9815bbd0 con 0x7fcb9815ef70 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47702/0 (socket says 172.21.15.70:47702) 2024-04-15T15:16:41.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/1928222502 learned_addr learned my addr 172.21.15.70:0/1928222502 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9d4b0640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb9809a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9e4b2640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 0x7fcb9809a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 msgr2=0x7fcb9809a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 0x7fcb9809a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 msgr2=0x7fcb9809a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb9809a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 -- 172.21.15.70:0/1928222502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb9809acd0 con 0x7fcb980b3160 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9e4b2640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 0x7fcb9809a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9dcb1640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009d90 tx=0x7fcb94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94039050 con 0x7fcb980b3160 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb94045070 con 0x7fcb980b3160 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94009450 con 0x7fcb980b3160 2024-04-15T15:16:41.479 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb9814a4b0 con 0x7fcb980b3160 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.474+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb9809b260 con 0x7fcb980b3160 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.475+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb980689e0 con 0x7fcb980b3160 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.476+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcb9403e610 con 0x7fcb980b3160 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.476+0000 7fcb8e7fc640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 0x7fcb7c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.476+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fcb940ae7e0 con 0x7fcb980b3160 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.476+0000 7fcb9d4b0640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 0x7fcb7c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.477+0000 7fcb9d4b0640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 0x7fcb7c06ee40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcb981f58c0 tx=0x7fcb88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.482 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.480+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcb9407cad0 con 0x7fcb980b3160 2024-04-15T15:16:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.652+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcb980739e0 con 0x7fcb980b3160 2024-04-15T15:16:41.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.653+0000 7fcb8e7fc640 1 -- 172.21.15.70:0/1928222502 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7fcb9407c3d0 con 0x7fcb980b3160 2024-04-15T15:16:41.655 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 msgr2=0x7fcb7c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 0x7fcb7c06ee40 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcb981f58c0 tx=0x7fcb88008040 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 msgr2=0x7fcb980aee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fcb94009d90 tx=0x7fcb94004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 shutdown_connections 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcb9815ef70 0x7fcb9809a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcb980d0690 0x7fcb9809a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcb7c06c980 0x7fcb7c06ee40 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 --2- 172.21.15.70:0/1928222502 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcb980b3160 0x7fcb980aee00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 >> 172.21.15.70:0/1928222502 conn(0x7fcb9806be00 msgr2=0x7fcb98074e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 shutdown_connections 2024-04-15T15:16:41.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.655+0000 7fcb9ff15640 1 -- 172.21.15.70:0/1928222502 wait complete. 2024-04-15T15:16:41.668 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:41.692 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:41.713 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.775+0000 7f45ecce4640 1 Processor -- start 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.775+0000 7f45ecce4640 1 -- start start 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.775+0000 7f45ecce4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f45e815e790 0x7f45e80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.775+0000 7f45ecce4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f45e80cfd20 0x7f45e80d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.775+0000 7f45ecce4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45e80d2480 con 0x7f45e80d0f00 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45ecce4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f45e80d2db0 con 0x7f45e815e790 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45ecce4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f45e80d2760 con 0x7f45e80cfd20 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f45e815e790 0x7f45e80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f45e815e790 0x7f45e80d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47718/0 (socket says 172.21.15.70:47718) 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 -- 172.21.15.70:0/2134965164 learned_addr learned my addr 172.21.15.70:0/2134965164 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e6d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f45e80d0f00 0x7f45e80b74f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44490/0 (socket says 172.21.15.70:44490) 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2134965164 >> v2:172.21.15.96:3300/0 conn(0x7f45e80cfd20 0x7f45e80d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 -- 172.21.15.70:0/2134965164 >> v2:172.21.15.96:3300/0 conn(0x7f45e80cfd20 msgr2=0x7f45e80d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 --2- 172.21.15.70:0/2134965164 >> v2:172.21.15.96:3300/0 conn(0x7f45e80cfd20 0x7f45e80d0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 -- 172.21.15.70:0/2134965164 >> v1:172.21.15.70:6789/0 conn(0x7f45e80d0f00 legacy=0x7f45e80b74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e559a640 1 -- 172.21.15.70:0/2134965164 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903129507 0 0) 0x7f45e80d2480 con 0x7f45e80d0f00 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.776+0000 7f45e659c640 1 -- 172.21.15.70:0/2134965164 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e80d2a40 con 0x7f45e815e790 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45e659c640 1 --2- 172.21.15.70:0/2134965164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e80d3240 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f45e808e510 tx=0x7f45d802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929760f8d663d4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45e559a640 1 -- 172.21.15.70:0/2134965164 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8034070 con 0x7f45e815e790 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45e559a640 1 -- 172.21.15.70:0/2134965164 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d8030860 con 0x7f45e815e790 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45e559a640 1 -- 172.21.15.70:0/2134965164 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d8030c30 con 0x7f45e815e790 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 -- 172.21.15.70:0/2134965164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 msgr2=0x7f45e80d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2134965164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e80d3240 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f45e808e510 tx=0x7f45d802b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 -- 172.21.15.70:0/2134965164 shutdown_connections 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2134965164 >> v2:172.21.15.96:3300/0 conn(0x7f45e80cfd20 0x7f45e80d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2134965164 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e80d3240 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 -- 172.21.15.70:0/2134965164 >> 172.21.15.70:0/2134965164 conn(0x7f45e808bf60 msgr2=0x7f45e80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.777+0000 7f45ecce4640 1 -- 172.21.15.70:0/2134965164 shutdown_connections 2024-04-15T15:16:41.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.778+0000 7f45ecce4640 1 -- 172.21.15.70:0/2134965164 wait complete. 2024-04-15T15:16:41.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.779+0000 7f45ecce4640 1 Processor -- start 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.779+0000 7f45ecce4640 1 -- start start 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 0x7f45e80cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e80d0f00 0x7f45e8156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45e815ecf0 con 0x7f45e815e790 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45e815eb70 con 0x7f45e80cfd20 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f45e80d13b0 con 0x7f45e80d0f00 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47730/0 (socket says 172.21.15.70:47730) 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2808336119 learned_addr learned my addr 172.21.15.70:0/2808336119 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e659c640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 0x7f45e80cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e5d9b640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e80d0f00 0x7f45e8156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e80d0f00 msgr2=0x7f45e8156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e80d0f00 0x7f45e8156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 msgr2=0x7f45e80cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 0x7f45e80cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 -- 172.21.15.70:0/2808336119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e809f880 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e659c640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 0x7f45e80cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45e6d9d640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c7a0 tx=0x7f45e000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e0018070 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45e000a640 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e809fb00 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e000aa70 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.780+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e80a00c0 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.781+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45e80689e0 con 0x7f45e815e790 2024-04-15T15:16:41.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.782+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f45e001b070 con 0x7f45e815e790 2024-04-15T15:16:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.782+0000 7f45d6ffd640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 0x7f45c806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.782+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f45e008b360 con 0x7f45e815e790 2024-04-15T15:16:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.782+0000 7f45e659c640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 0x7f45c806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:41.787 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.783+0000 7f45e659c640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 0x7f45c806ee40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f45d8036000 tx=0x7f45d8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:41.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.787+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45e809f880 con 0x7f45e815e790 2024-04-15T15:16:41.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.965+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45e80d1ce0 con 0x7f45e815e790 2024-04-15T15:16:41.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.966+0000 7f45d6ffd640 1 -- 172.21.15.70:0/2808336119 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f45e0059610 con 0x7f45e815e790 2024-04-15T15:16:41.968 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 msgr2=0x7f45c806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 0x7f45c806ee40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f45d8036000 tx=0x7f45d8008030 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 msgr2=0x7f45e809f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f45e000c7a0 tx=0x7f45e000cc60 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 shutdown_connections 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45c806c980 0x7f45c806ee40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e815e790 0x7f45e809f340 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e80d0f00 0x7f45e8156ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 --2- 172.21.15.70:0/2808336119 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e80cfd20 0x7f45e80cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 >> 172.21.15.70:0/2808336119 conn(0x7f45e808bf60 msgr2=0x7f45e80d4680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:41.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 shutdown_connections 2024-04-15T15:16:41.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:41.968+0000 7f45ecce4640 1 -- 172.21.15.70:0/2808336119 wait complete. 2024-04-15T15:16:41.983 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:42.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.130+0000 7ff4057c8640 1 Processor -- start 2024-04-15T15:16:42.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.130+0000 7ff4057c8640 1 -- start start 2024-04-15T15:16:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff4057c8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff4000d3270 0x7ff4000cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff4057c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4000b47a0 con 0x7ff40015e700 2024-04-15T15:16:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff4057c8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4000b4a80 con 0x7ff4000d2360 2024-04-15T15:16:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff4057c8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff4000b53b0 con 0x7ff4000d3270 2024-04-15T15:16:42.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3feffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff40015e700 0x7ff4000b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44494/0 (socket says 172.21.15.70:44494) 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3feffd640 1 -- 172.21.15.70:0/2948273495 learned_addr learned my addr 172.21.15.70:0/2948273495 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2299841761 0 0) 0x7ff4000b47a0 con 0x7ff40015e700 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3ec003520 con 0x7ff40015e700 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3ff7fe640 1 --2- 172.21.15.70:0/2948273495 >> v2:172.21.15.96:3300/0 conn(0x7ff4000d3270 0x7ff4000cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3863421952 0 0) 0x7ff3ec003520 con 0x7ff40015e700 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 >> v2:172.21.15.96:3300/0 conn(0x7ff4000d3270 msgr2=0x7ff4000cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 --2- 172.21.15.70:0/2948273495 >> v2:172.21.15.96:3300/0 conn(0x7ff4000d3270 0x7ff4000cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 >> v1:172.21.15.96:6789/0 conn(0x7ff4000d2360 legacy=0x7ff4000d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4000b4d60 con 0x7ff40015e700 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7ff3f00017f0 con 0x7ff40015e700 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3ff7fe640 1 --2- 172.21.15.70:0/2948273495 >> v2:172.21.15.96:3300/0 conn(0x7ff4000d3270 0x7ff4000cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 >> v1:172.21.15.70:6789/0 conn(0x7ff40015e700 legacy=0x7ff4000b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff3ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff3ec005210 0x7ff3ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4000b53b0 con 0x7ff3ec004800 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff3ec00c140 con 0x7ff3ec0039b0 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff3ec00c340 con 0x7ff3ec005210 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff3ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3feffd640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.131+0000 7ff3ff7fe640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff3ec005210 0x7ff3ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 -- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff3ec005210 msgr2=0x7ff3ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff3ec005210 0x7ff3ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 -- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3ec0039b0 msgr2=0x7ff3ec003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff3ec0039b0 0x7ff3ec003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 -- 172.21.15.70:0/2948273495 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3ec003520 con 0x7ff3ec004800 2024-04-15T15:16:42.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3ff7fe640 1 -- 172.21.15.70:0/2948273495 reap_dead start 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff3ec004c80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8001750 tx=0x7ff3ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8223194b141901b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8001370 con 0x7ff3ec004800 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3e80052d0 con 0x7ff3ec004800 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.132+0000 7ff3fe7fc640 1 -- 172.21.15.70:0/2948273495 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3e8003830 con 0x7ff3ec004800 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 -- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 msgr2=0x7ff3ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff3ec004c80 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8001750 tx=0x7ff3ec005830 comp rx=0 tx=0).stop 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 -- 172.21.15.70:0/2948273495 shutdown_connections 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2948273495 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff3ec004c80 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 -- 172.21.15.70:0/2948273495 >> 172.21.15.70:0/2948273495 conn(0x7ff40008bf60 msgr2=0x7ff4000d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 -- 172.21.15.70:0/2948273495 shutdown_connections 2024-04-15T15:16:42.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.133+0000 7ff4057c8640 1 -- 172.21.15.70:0/2948273495 wait complete. 2024-04-15T15:16:42.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.134+0000 7ff4057c8640 1 Processor -- start 2024-04-15T15:16:42.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.134+0000 7ff4057c8640 1 -- start start 2024-04-15T15:16:42.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff40014eae0 0x7ff40014e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4000b9440 0x7ff40014b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff4000d1d40 con 0x7ff3ec004800 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff4000b4450 con 0x7ff4000b9440 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff4057c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff4000d1ec0 con 0x7ff40014eae0 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47746/0 (socket says 172.21.15.70:47746) 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 -- 172.21.15.70:0/2440471 learned_addr learned my addr 172.21.15.70:0/2440471 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3ff7fe640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4000b9440 0x7ff40014b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff40014eae0 0x7ff40014e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 -- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff40014eae0 msgr2=0x7ff40014e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff40014eae0 0x7ff40014e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 -- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4000b9440 msgr2=0x7ff40014b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4000b9440 0x7ff40014b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.135+0000 7ff3feffd640 1 -- 172.21.15.70:0/2440471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4000ce3f0 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff3feffd640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e7b0 tx=0x7ff3f000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000ee60 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3f0018070 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4000b8c80 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3f0007500 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.136+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4000ce970 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.137+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4000ce7a0 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.137+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff3f00057a0 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.138+0000 7ff3f7fff640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 0x7ff3e006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.138+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff3f008bc30 con 0x7ff3ec004800 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.138+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 0x7ff3e006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.138+0000 7ff3f67fc640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 0x7ff3e006ee70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8033000 tx=0x7ff3e8009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.142+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff3f0059e50 con 0x7ff3ec004800 2024-04-15T15:16:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.318+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff4000689e0 con 0x7ff3ec004800 2024-04-15T15:16:42.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.318+0000 7ff3f7fff640 1 -- 172.21.15.70:0/2440471 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7ff3f0059750 con 0x7ff3ec004800 2024-04-15T15:16:42.320 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:42.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 msgr2=0x7ff3e006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 0x7ff3e006ee70 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7ff3e8033000 tx=0x7ff3e8009030 comp rx=0 tx=0).stop 2024-04-15T15:16:42.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 msgr2=0x7ff4000cb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e7b0 tx=0x7ff3f000ec70 comp rx=0 tx=0).stop 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 shutdown_connections 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff4000b9440 0x7ff40014b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff40014eae0 0x7ff40014e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff3e006c9b0 0x7ff3e006ee70 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 --2- 172.21.15.70:0/2440471 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff3ec004800 0x7ff4000cb650 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 >> 172.21.15.70:0/2440471 conn(0x7ff40008bf60 msgr2=0x7ff4000b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 shutdown_connections 2024-04-15T15:16:42.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.320+0000 7ff4057c8640 1 -- 172.21.15.70:0/2440471 wait complete. 2024-04-15T15:16:42.333 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.435+0000 7f667bceb640 1 Processor -- start 2024-04-15T15:16:42.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.435+0000 7f667bceb640 1 -- start start 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667bceb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f667415fda0 0x7f66740ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667bceb640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f66740cf800 0x7f66740cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667bceb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66740b1f40 con 0x7f66740ced30 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667bceb640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f66740d0150 con 0x7f66740cf800 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667bceb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f66740cda30 con 0x7f667415fda0 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667a288640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f66740cf800 0x7f66740cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f6679286640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f66740ced30 0x7f66740b1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48288/0 (socket says 172.21.15.70:48288) 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f6679286640 1 -- 172.21.15.70:0/568116426 learned_addr learned my addr 172.21.15.70:0/568116426 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f6679a87640 1 --2- 172.21.15.70:0/568116426 >> v2:172.21.15.96:3300/0 conn(0x7f667415fda0 0x7f66740ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667a288640 1 -- 172.21.15.70:0/568116426 >> v2:172.21.15.96:3300/0 conn(0x7f667415fda0 msgr2=0x7f66740ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667a288640 1 --2- 172.21.15.70:0/568116426 >> v2:172.21.15.96:3300/0 conn(0x7f667415fda0 0x7f66740ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667a288640 1 -- 172.21.15.70:0/568116426 >> v1:172.21.15.70:6789/0 conn(0x7f66740ced30 legacy=0x7f66740b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.436+0000 7f667a288640 1 -- 172.21.15.70:0/568116426 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66740cdd10 con 0x7f66740cf800 2024-04-15T15:16:42.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.437+0000 7f667a288640 1 --2- 172.21.15.70:0/568116426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740cf800 0x7f66740cfc10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f66740d2390 tx=0x7f666c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b0129d326dbd0b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.437+0000 7f6678a85640 1 -- 172.21.15.70:0/568116426 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c039070 con 0x7f66740cf800 2024-04-15T15:16:42.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.437+0000 7f6678a85640 1 -- 172.21.15.70:0/568116426 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666c02cc80 con 0x7f66740cf800 2024-04-15T15:16:42.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.437+0000 7f6678a85640 1 -- 172.21.15.70:0/568116426 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c034660 con 0x7f66740cf800 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 -- 172.21.15.70:0/568116426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740cf800 msgr2=0x7f66740cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 --2- 172.21.15.70:0/568116426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740cf800 0x7f66740cfc10 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f66740d2390 tx=0x7f666c02bed0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 -- 172.21.15.70:0/568116426 shutdown_connections 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 --2- 172.21.15.70:0/568116426 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740cf800 0x7f66740cfc10 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 --2- 172.21.15.70:0/568116426 >> v2:172.21.15.96:3300/0 conn(0x7f667415fda0 0x7f66740ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 -- 172.21.15.70:0/568116426 >> 172.21.15.70:0/568116426 conn(0x7f66740b6870 msgr2=0x7f66740b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 -- 172.21.15.70:0/568116426 shutdown_connections 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.438+0000 7f667bceb640 1 -- 172.21.15.70:0/568116426 wait complete. 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.439+0000 7f667bceb640 1 Processor -- start 2024-04-15T15:16:42.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.439+0000 7f667bceb640 1 -- start start 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.439+0000 7f667bceb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66740cf800 0x7f667415c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f667415fda0 0x7f667415dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6674160180 con 0x7f66740ced30 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f66740ce2d0 con 0x7f66740cf800 2024-04-15T15:16:42.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f66740b0790 con 0x7f667415fda0 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49840/0 (socket says 172.21.15.70:49840) 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 -- 172.21.15.70:0/717246958 learned_addr learned my addr 172.21.15.70:0/717246958 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 -- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f667415fda0 msgr2=0x7f667415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f667415fda0 0x7f667415dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 -- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66740cf800 msgr2=0x7f667415c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66740cf800 0x7f667415c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 -- 172.21.15.70:0/717246958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667415ca20 con 0x7f66740ced30 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f6679a87640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f667000c820 tx=0x7f667000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670015070 con 0x7f66740ced30 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6670004240 con 0x7f66740ced30 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667415cd00 con 0x7f66740ced30 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667000ce10 con 0x7f66740ced30 2024-04-15T15:16:42.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.440+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f667415d2c0 con 0x7f66740ced30 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.441+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66740689e0 con 0x7f66740ced30 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.442+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6670007830 con 0x7f66740ced30 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.442+0000 7f666a7fc640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 0x7f665806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.442+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f667008bd40 con 0x7f66740ced30 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.442+0000 7f6679286640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 0x7f665806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:42.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.443+0000 7f6679286640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 0x7f665806ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6664000f60 tx=0x7f6664007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:42.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.446+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6670059ee0 con 0x7f66740ced30 2024-04-15T15:16:42.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.635+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6674154510 con 0x7f66740ced30 2024-04-15T15:16:42.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.635+0000 7f666a7fc640 1 -- 172.21.15.70:0/717246958 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f66700597e0 con 0x7f66740ced30 2024-04-15T15:16:42.637 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 msgr2=0x7f665806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 0x7f665806ee40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6664000f60 tx=0x7f6664007a80 comp rx=0 tx=0).stop 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 msgr2=0x7f6674153ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f667000c820 tx=0x7f667000cce0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 shutdown_connections 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f667415fda0 0x7f667415dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f66740cf800 0x7f667415c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f665806c980 0x7f665806ee40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.639 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 --2- 172.21.15.70:0/717246958 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66740ced30 0x7f6674153ee0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:42.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 >> 172.21.15.70:0/717246958 conn(0x7f66740b6870 msgr2=0x7f66740b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:42.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 shutdown_connections 2024-04-15T15:16:42.640 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:42.637+0000 7f667bceb640 1 -- 172.21.15.70:0/717246958 wait complete. 2024-04-15T15:16:42.650 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:16:42.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:42.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:42.677 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[53692]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:42.678 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:48.780 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:16:48.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:48.781 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:16:48.801 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:48.801 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:48.821 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:48.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.953+0000 7f4d9aa8f640 1 Processor -- start 2024-04-15T15:16:48.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.953+0000 7f4d9aa8f640 1 -- start start 2024-04-15T15:16:48.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9aa8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d9415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9aa8f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d9415cb50 con 0x7f4d940d0690 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9aa8f640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d9415bfd0 con 0x7f4d940ae280 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9aa8f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9415c2b0 con 0x7f4d940b3160 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9882b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4d940d0690 0x7f4d940d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48296/0 (socket says 172.21.15.70:48296) 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d9882b640 1 -- 172.21.15.70:0/2377788462 learned_addr learned my addr 172.21.15.70:0/2377788462 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d93fff640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d9415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3794417112 0 0) 0x7f4d9415cb50 con 0x7f4d940d0690 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.954+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d84005730 con 0x7f4d940d0690 2024-04-15T15:16:48.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 97893963 0 0) 0x7f4d9415bfd0 con 0x7f4d940ae280 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d9415cb50 con 0x7f4d940ae280 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2625283240 0 0) 0x7f4d84005730 con 0x7f4d940d0690 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 msgr2=0x7f4d9415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d9415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 >> v1:172.21.15.96:6789/0 conn(0x7f4d940ae280 legacy=0x7f4d940ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d9415c590 con 0x7f4d940d0690 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4d88001670 con 0x7f4d940d0690 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 >> v1:172.21.15.70:6789/0 conn(0x7f4d940d0690 legacy=0x7f4d940d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d84005d80 0x7f4d84006180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d84006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d840073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9415c2b0 con 0x7f4d84006a10 2024-04-15T15:16:48.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400dc80 con 0x7f4d84005d80 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4d8400de50 con 0x7f4d940b3160 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d84006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d9882b640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d84005d80 0x7f4d84006180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d9902c640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d840073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 -- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 msgr2=0x7f4d840073d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d940b3160 0x7f4d840073d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 -- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d84005d80 msgr2=0x7f4d84006180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d84005d80 0x7f4d84006180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d93fff640 1 -- 172.21.15.70:0/2377788462 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d84005730 con 0x7f4d84006a10 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.955+0000 7f4d9882b640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d84005d80 0x7f4d84006180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d93fff640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d84006e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009ac0 tx=0x7f4d84007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6562a3e2cbcf81 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80039070 con 0x7f4d84006a10 2024-04-15T15:16:48.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d8002c660 con 0x7f4d84006a10 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d937fe640 1 -- 172.21.15.70:0/2377788462 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d80009690 con 0x7f4d84006a10 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/2377788462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 msgr2=0x7f4d84006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/2377788462 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d84006e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009ac0 tx=0x7f4d84007a40 comp rx=0 tx=0).stop 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.956+0000 7f4d9902c640 1 -- 172.21.15.70:0/2377788462 reap_dead start 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.957+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/2377788462 shutdown_connections 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.957+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/2377788462 >> 172.21.15.70:0/2377788462 conn(0x7f4d9406be00 msgr2=0x7f4d9406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.957+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/2377788462 shutdown_connections 2024-04-15T15:16:48.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.957+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/2377788462 wait complete. 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.958+0000 7f4d9aa8f640 1 Processor -- start 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.958+0000 7f4d9aa8f640 1 -- start start 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d940b0820 0x7f4d940cee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d94155d90 0x7f4d941592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4d9415b690 con 0x7f4d84006a10 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4d940b4fb0 con 0x7f4d940b0820 2024-04-15T15:16:48.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9aa8f640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4d9415b810 con 0x7f4d94155d90 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49862/0 (socket says 172.21.15.70:49862) 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 -- 172.21.15.70:0/826972976 learned_addr learned my addr 172.21.15.70:0/826972976 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d93fff640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d940b0820 0x7f4d940cee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9902c640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d94155d90 0x7f4d941592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 -- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d94155d90 msgr2=0x7f4d941592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d94155d90 0x7f4d941592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 -- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d940b0820 msgr2=0x7f4d940cee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d940b0820 0x7f4d940cee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 -- 172.21.15.70:0/826972976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d94154fd0 con 0x7f4d84006a10 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d9882b640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800e870 tx=0x7f4d8800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8800c8d0 con 0x7f4d84006a10 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.959+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d88018070 con 0x7f4d84006a10 2024-04-15T15:16:48.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.960+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d8800cc90 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.960+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d941552b0 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.960+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d94155870 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.960+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d94072cc0 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.961+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4d8801d460 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.961+0000 7f4d917fa640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 0x7f4d7806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.961+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4d8808bb20 con 0x7f4d84006a10 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.961+0000 7f4d93fff640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 0x7f4d7806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.962+0000 7f4d93fff640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 0x7f4d7806ee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4d80036000 tx=0x7f4d80030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:48.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:48.966+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4d94072cc0 con 0x7f4d84006a10 2024-04-15T15:16:49.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.144+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4d940689e0 con 0x7f4d84006a10 2024-04-15T15:16:49.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.145+0000 7f4d917fa640 1 -- 172.21.15.70:0/826972976 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16882 (secure 0 0 0) 0x7f4d88059810 con 0x7f4d84006a10 2024-04-15T15:16:49.147 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:49.147 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:26.878225+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 msgr2=0x7f4d7806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 0x7f4d7806ee70 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4d80036000 tx=0x7f4d80030000 comp rx=0 tx=0).stop 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 msgr2=0x7f4d941f00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f4d8800e870 tx=0x7f4d8800ed30 comp rx=0 tx=0).stop 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 shutdown_connections 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4d94155d90 0x7f4d941592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4d940b0820 0x7f4d940cee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4d7806c9b0 0x7f4d7806ee70 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 --2- 172.21.15.70:0/826972976 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4d84006a10 0x7f4d941f00e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 >> 172.21.15.70:0/826972976 conn(0x7f4d9406be00 msgr2=0x7f4d940720b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 shutdown_connections 2024-04-15T15:16:49.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.147+0000 7f4d9aa8f640 1 -- 172.21.15.70:0/826972976 wait complete. 2024-04-15T15:16:49.160 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:49.161 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.265+0000 7f88651cd640 1 Processor -- start 2024-04-15T15:16:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.265+0000 7f88651cd640 1 -- start start 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f88651cd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f886015bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f88651cd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f88600b5230 0x7f8860072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f88651cd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f886015c1d0 con 0x7f8860072c60 2024-04-15T15:16:49.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f88651cd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f88600b4da0 con 0x7f88600b5230 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f88651cd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f88600b4190 con 0x7f8860158770 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885f59e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8860072c60 0x7f886015e800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48308/0 (socket says 172.21.15.70:48308) 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885f59e640 1 -- 172.21.15.70:0/3545361027 learned_addr learned my addr 172.21.15.70:0/3545361027 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3527431297 0 0) 0x7f8858001490 con 0x7f8860072c60 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8848003520 con 0x7f8860072c60 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885e59c640 1 --2- 172.21.15.70:0/3545361027 >> v2:172.21.15.96:3300/0 conn(0x7f88600b5230 0x7f8860072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.266+0000 7f885ed9d640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f886015bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1804634041 0 0) 0x7f8848003520 con 0x7f8860072c60 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 msgr2=0x7f886015bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f886015bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 >> v2:172.21.15.96:3300/0 conn(0x7f88600b5230 msgr2=0x7f8860072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 --2- 172.21.15.70:0/3545361027 >> v2:172.21.15.96:3300/0 conn(0x7f88600b5230 0x7f8860072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88600b4470 con 0x7f8860072c60 2024-04-15T15:16:49.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f8858001bc0 con 0x7f8860072c60 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 >> v1:172.21.15.70:6789/0 conn(0x7f8860072c60 legacy=0x7f886015e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f88480039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848003f30 0x7f8848004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8848004d40 0x7f88480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88600b4da0 con 0x7f8848004d40 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88600b4190 con 0x7f8848003f30 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f884800b9c0 con 0x7f8860158770 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8848004d40 0x7f88480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885ed9d640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f88480039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885e59c640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848003f30 0x7f8848004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 -- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 msgr2=0x7f88480039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8860158770 0x7f88480039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 -- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848003f30 msgr2=0x7f8848004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848003f30 0x7f8848004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.267+0000 7f885f59e640 1 -- 172.21.15.70:0/3545361027 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8848003520 con 0x7f8848004d40 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f885f59e640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8848004d40 0x7f88480051c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f885800aa50 tx=0x7f8848005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fec88c0845b0fa9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8858038070 con 0x7f8848004d40 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8858033650 con 0x7f8848004d40 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f885dd9b640 1 -- 172.21.15.70:0/3545361027 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885800a670 con 0x7f8848004d40 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f88651cd640 1 -- 172.21.15.70:0/3545361027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8848004d40 msgr2=0x7f88480051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.268+0000 7f88651cd640 1 --2- 172.21.15.70:0/3545361027 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8848004d40 0x7f88480051c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f885800aa50 tx=0x7f8848005830 comp rx=0 tx=0).stop 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.269+0000 7f885f59e640 1 -- 172.21.15.70:0/3545361027 reap_dead start 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.269+0000 7f88651cd640 1 -- 172.21.15.70:0/3545361027 shutdown_connections 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.269+0000 7f88651cd640 1 -- 172.21.15.70:0/3545361027 >> 172.21.15.70:0/3545361027 conn(0x7f886006be00 msgr2=0x7f886006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.269+0000 7f88651cd640 1 -- 172.21.15.70:0/3545361027 shutdown_connections 2024-04-15T15:16:49.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.269+0000 7f88651cd640 1 -- 172.21.15.70:0/3545361027 wait complete. 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.270+0000 7f88651cd640 1 Processor -- start 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.270+0000 7f88651cd640 1 -- start start 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 0x7f88601f29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f88601f3be0 0x7f8860150a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8860071fa0 con 0x7f8860154da0 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f88600b4a30 con 0x7f8848004d40 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f88651cd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8860158b50 con 0x7f88601f3be0 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49874/0 (socket says 172.21.15.70:49874) 2024-04-15T15:16:49.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 -- 172.21.15.70:0/175293959 learned_addr learned my addr 172.21.15.70:0/175293959 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885ed9d640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 0x7f88601f29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 -- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f88601f3be0 msgr2=0x7f8860150a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f88601f3be0 0x7f8860150a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 -- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 msgr2=0x7f88601f29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 0x7f88601f29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 -- 172.21.15.70:0/175293959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88601f31e0 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885ed9d640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 0x7f88601f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.271+0000 7f885e59c640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8854011980 tx=0x7f8854011e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.272+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885400b040 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.272+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8854004440 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.272+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88601f34c0 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.272+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f885400fb40 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.272+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88601f2ee0 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.273+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88601f39e0 con 0x7f8860154da0 2024-04-15T15:16:49.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.273+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f885400fce0 con 0x7f8860154da0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.274+0000 7f88477fe640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 0x7f884c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.274+0000 7f885ed9d640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 0x7f884c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.274+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f885408c6d0 con 0x7f8860154da0 2024-04-15T15:16:49.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.274+0000 7f885ed9d640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 0x7f884c06ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8860156dc0 tx=0x7f8850008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.278+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f88601f39e0 con 0x7f8860154da0 2024-04-15T15:16:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.452+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f88600689e0 con 0x7f8860154da0 2024-04-15T15:16:49.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.452+0000 7f88477fe640 1 -- 172.21.15.70:0/175293959 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f885405aa10 con 0x7f8860154da0 2024-04-15T15:16:49.455 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:49.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.454+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 msgr2=0x7f884c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.454+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 0x7f884c06ee40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8860156dc0 tx=0x7f8850008040 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.454+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 msgr2=0x7f88601525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.454+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f8854011980 tx=0x7f8854011e40 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 shutdown_connections 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f88601f3be0 0x7f8860150a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f884c06c980 0x7f884c06ee40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8860154da0 0x7f88601525d0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 --2- 172.21.15.70:0/175293959 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8848004d40 0x7f88601f29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 >> 172.21.15.70:0/175293959 conn(0x7f886006be00 msgr2=0x7f88600b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 shutdown_connections 2024-04-15T15:16:49.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.455+0000 7f88651cd640 1 -- 172.21.15.70:0/175293959 wait complete. 2024-04-15T15:16:49.468 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.572+0000 7f3a128c5640 1 Processor -- start 2024-04-15T15:16:49.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.572+0000 7f3a128c5640 1 -- start start 2024-04-15T15:16:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a128c5640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3a0c15e830 0x7f3a0c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a128c5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3a0c15ec10 0x7f3a0c067720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a128c5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a0c0b3c10 con 0x7f3a0c067c60 2024-04-15T15:16:49.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a128c5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f3a0c0b3ef0 con 0x7f3a0c15ec10 2024-04-15T15:16:49.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a128c5640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3a0c0b41d0 con 0x7f3a0c15e830 2024-04-15T15:16:49.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3a0c15ec10 0x7f3a0c067720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f3a0c15ec10 0x7f3a0c067720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49886/0 (socket says 172.21.15.70:49886) 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 -- 172.21.15.70:0/4005032145 learned_addr learned my addr 172.21.15.70:0/4005032145 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0bfff640 1 --2- 172.21.15.70:0/4005032145 >> v2:172.21.15.96:3300/0 conn(0x7f3a0c15e830 0x7f3a0c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 -- 172.21.15.70:0/4005032145 >> v2:172.21.15.96:3300/0 conn(0x7f3a0c15e830 msgr2=0x7f3a0c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 --2- 172.21.15.70:0/4005032145 >> v2:172.21.15.96:3300/0 conn(0x7f3a0c15e830 0x7f3a0c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 -- 172.21.15.70:0/4005032145 >> v1:172.21.15.96:6789/0 conn(0x7f3a0c067c60 legacy=0x7f3a0c0b53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.573+0000 7f3a0b7fe640 1 -- 172.21.15.70:0/4005032145 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c0b44b0 con 0x7f3a0c15ec10 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.574+0000 7f3a0b7fe640 1 --2- 172.21.15.70:0/4005032145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c15ec10 0x7f3a0c067720 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c073d70 tx=0x7f3a0002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12cb96771cd8b854 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.574+0000 7f3a0affd640 1 -- 172.21.15.70:0/4005032145 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0002e070 con 0x7f3a0c15ec10 2024-04-15T15:16:49.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.574+0000 7f3a0affd640 1 -- 172.21.15.70:0/4005032145 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a00037070 con 0x7f3a0c15ec10 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.574+0000 7f3a0affd640 1 -- 172.21.15.70:0/4005032145 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a00031a00 con 0x7f3a0c15ec10 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 -- 172.21.15.70:0/4005032145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c15ec10 msgr2=0x7f3a0c067720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 --2- 172.21.15.70:0/4005032145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c15ec10 0x7f3a0c067720 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c073d70 tx=0x7f3a0002b880 comp rx=0 tx=0).stop 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 -- 172.21.15.70:0/4005032145 shutdown_connections 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 --2- 172.21.15.70:0/4005032145 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c15ec10 0x7f3a0c067720 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 --2- 172.21.15.70:0/4005032145 >> v2:172.21.15.96:3300/0 conn(0x7f3a0c15e830 0x7f3a0c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 -- 172.21.15.70:0/4005032145 >> 172.21.15.70:0/4005032145 conn(0x7f3a0c073360 msgr2=0x7f3a0c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 -- 172.21.15.70:0/4005032145 shutdown_connections 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.575+0000 7f3a128c5640 1 -- 172.21.15.70:0/4005032145 wait complete. 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.576+0000 7f3a128c5640 1 Processor -- start 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.576+0000 7f3a128c5640 1 -- start start 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a0c067c60 0x7f3a0c143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a0c15e830 0x7f3a0c1e9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c1600e0 con 0x7f3a0c1e6790 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a0c15ff60 con 0x7f3a0c15e830 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a128c5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0c0b5ad0 con 0x7f3a0c067c60 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49896/0 (socket says 172.21.15.70:49896) 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 -- 172.21.15.70:0/551744751 learned_addr learned my addr 172.21.15.70:0/551744751 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a0b7fe640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a0c15e830 0x7f3a0c1e9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a0bfff640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a0c067c60 0x7f3a0c143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 -- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a0c067c60 msgr2=0x7f3a0c143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a0c067c60 0x7f3a0c143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 -- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a0c15e830 msgr2=0x7f3a0c1e9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a0c15e830 0x7f3a0c1e9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.577+0000 7f3a10e62640 1 -- 172.21.15.70:0/551744751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a0c143a70 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a10e62640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c780 tx=0x7f3a0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a04004b60 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a04004d00 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a0c143d50 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0400aa90 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.578+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a0c1e97b0 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.579+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0c0689e0 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.579+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3a0400ac30 con 0x7f3a0c1e6790 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.580+0000 7f3a08ff9640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 0x7f39f806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.580+0000 7f3a0bfff640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 0x7f39f806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.580+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f3a0408bfb0 con 0x7f3a0c1e6790 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.580+0000 7f3a0bfff640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 0x7f39f806ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006fd0 tx=0x7f39fc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.584+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a04020070 con 0x7f3a0c1e6790 2024-04-15T15:16:49.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.757+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3a0c06ff10 con 0x7f3a0c1e6790 2024-04-15T15:16:49.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.758+0000 7f3a08ff9640 1 -- 172.21.15.70:0/551744751 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f3a0405a260 con 0x7f3a0c1e6790 2024-04-15T15:16:49.759 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:49.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.759+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 msgr2=0x7f39f806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.759+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 0x7f39f806ee40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f39fc006fd0 tx=0x7f39fc008040 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.759+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 msgr2=0x7f3a0c158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.759+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c780 tx=0x7f3a0400cc40 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 shutdown_connections 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f39f806c980 0x7f39f806ee40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a0c1e6790 0x7f3a0c158670 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a0c15e830 0x7f3a0c1e9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 --2- 172.21.15.70:0/551744751 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a0c067c60 0x7f3a0c143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 >> 172.21.15.70:0/551744751 conn(0x7f3a0c073360 msgr2=0x7f3a0c15b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 shutdown_connections 2024-04-15T15:16:49.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.760+0000 7f3a128c5640 1 -- 172.21.15.70:0/551744751 wait complete. 2024-04-15T15:16:49.773 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:49.879 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.877+0000 7fbb08152640 1 Processor -- start 2024-04-15T15:16:49.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.878+0000 7fbb08152640 1 -- start start 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.878+0000 7fbb08152640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb0009a9d0 0x7fbb00160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.878+0000 7fbb08152640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.878+0000 7fbb08152640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb00070720 con 0x7fbb000701f0 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb08152640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbb0009ba90 con 0x7fbb0009a9d0 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb08152640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbb000d08c0 con 0x7fbb0015fb30 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb05eee640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbb0009a9d0 0x7fbb00160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb056ed640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbb000701f0 0x7fbb0009adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48320/0 (socket says 172.21.15.70:48320) 2024-04-15T15:16:49.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb056ed640 1 -- 172.21.15.70:0/3021690645 learned_addr learned my addr 172.21.15.70:0/3021690645 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb066ef640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb05eee640 1 -- 172.21.15.70:0/3021690645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 msgr2=0x7fbb000d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb05eee640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb05eee640 1 -- 172.21.15.70:0/3021690645 >> v1:172.21.15.70:6789/0 conn(0x7fbb000701f0 legacy=0x7fbb0009adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb04eec640 1 -- 172.21.15.70:0/3021690645 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023658969 0 0) 0x7fbb00070720 con 0x7fbb000701f0 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb05eee640 1 -- 172.21.15.70:0/3021690645 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb00070a90 con 0x7fbb0009a9d0 2024-04-15T15:16:49.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.879+0000 7fbb066ef640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:49.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.880+0000 7fbb05eee640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0009a9d0 0x7fbb00160420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097a0 tx=0x7fbaf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c5a8d6f08a31ba server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.880+0000 7fbb04eec640 1 -- 172.21.15.70:0/3021690645 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4035020 con 0x7fbb0009a9d0 2024-04-15T15:16:49.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.880+0000 7fbb04eec640 1 -- 172.21.15.70:0/3021690645 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf4031830 con 0x7fbb0009a9d0 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.880+0000 7fbb04eec640 1 -- 172.21.15.70:0/3021690645 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4031c00 con 0x7fbb0009a9d0 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 -- 172.21.15.70:0/3021690645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0009a9d0 msgr2=0x7fbb00160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0009a9d0 0x7fbb00160420 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fbaf40097a0 tx=0x7fbaf402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 -- 172.21.15.70:0/3021690645 shutdown_connections 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 --2- 172.21.15.70:0/3021690645 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0009a9d0 0x7fbb00160420 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 -- 172.21.15.70:0/3021690645 >> 172.21.15.70:0/3021690645 conn(0x7fbb000d2460 msgr2=0x7fbb000d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 -- 172.21.15.70:0/3021690645 shutdown_connections 2024-04-15T15:16:49.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.881+0000 7fbb08152640 1 -- 172.21.15.70:0/3021690645 wait complete. 2024-04-15T15:16:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.882+0000 7fbb08152640 1 Processor -- start 2024-04-15T15:16:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.882+0000 7fbb08152640 1 -- start start 2024-04-15T15:16:49.886 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb000701f0 0x7fbb000b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0009a9d0 0x7fbb0014f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0015ffc0 con 0x7fbb0015fb30 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbb00071050 con 0x7fbb0009a9d0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb08152640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbb000b1c60 con 0x7fbb000701f0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb066ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb066ef640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49920/0 (socket says 172.21.15.70:49920) 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb066ef640 1 -- 172.21.15.70:0/262265319 learned_addr learned my addr 172.21.15.70:0/262265319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb05eee640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb000701f0 0x7fbb000b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.883+0000 7fbb056ed640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0009a9d0 0x7fbb0014f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 -- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb000701f0 msgr2=0x7fbb000b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb000701f0 0x7fbb000b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 -- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0009a9d0 msgr2=0x7fbb0014f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:49.887 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0009a9d0 0x7fbb0014f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 -- 172.21.15.70:0/262265319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0014fe10 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb066ef640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8007df0 tx=0x7fbaf800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf801a070 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaf8003990 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8003d80 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb000b7d80 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.884+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb000b08c0 con 0x7fbb0015fb30 2024-04-15T15:16:49.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.885+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb000689e0 con 0x7fbb0015fb30 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.885+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbaf801d070 con 0x7fbb0015fb30 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.886+0000 7fbaf27fc640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 0x7fbae406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.886+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbaf808c040 con 0x7fbb0015fb30 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.886+0000 7fbb05eee640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 0x7fbae406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:49.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.886+0000 7fbb05eee640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 0x7fbae406ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:49.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:49.891+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbb0014fe10 con 0x7fbb0015fb30 2024-04-15T15:16:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.066+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbb000b0fc0 con 0x7fbb0015fb30 2024-04-15T15:16:50.068 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.067+0000 7fbaf27fc640 1 -- 172.21.15.70:0/262265319 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v39) ==== 100+0+11 (secure 0 0 0) 0x7fbaf8010070 con 0x7fbb0015fb30 2024-04-15T15:16:50.069 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 msgr2=0x7fbae406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 0x7fbae406ee40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4036000 tx=0x7fbaf4008030 comp rx=0 tx=0).stop 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 msgr2=0x7fbb000d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8007df0 tx=0x7fbaf800d4a0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 shutdown_connections 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbae406c980 0x7fbae406ee40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbb0015fb30 0x7fbb000d0d50 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbb0009a9d0 0x7fbb0014f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 --2- 172.21.15.70:0/262265319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbb000701f0 0x7fbb000b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 >> 172.21.15.70:0/262265319 conn(0x7fbb000d2460 msgr2=0x7fbb0006caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 shutdown_connections 2024-04-15T15:16:50.072 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.069+0000 7fbb08152640 1 -- 172.21.15.70:0/262265319 wait complete. 2024-04-15T15:16:50.082 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:50.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.184+0000 7f8b4111c640 1 Processor -- start 2024-04-15T15:16:50.186 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.184+0000 7f8b4111c640 1 -- start start 2024-04-15T15:16:50.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b4111c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8b3c15c660 0x7f8b3c06e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b4111c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b3c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b4111c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b3c06eaf0 con 0x7f8b3c15cf80 2024-04-15T15:16:50.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b4111c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8b3c06edd0 con 0x7f8b3c15c660 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b4111c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3c06f0b0 con 0x7f8b3c0ccb20 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b3b59e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8b3c15cf80 0x7f8b3c0cfa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48332/0 (socket says 172.21.15.70:48332) 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b3b59e640 1 -- 172.21.15.70:0/909732235 learned_addr learned my addr 172.21.15.70:0/909732235 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b3c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/909732235 >> v2:172.21.15.96:3300/0 conn(0x7f8b3c15c660 0x7f8b3c06e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1527745243 0 0) 0x7f8b3c06eaf0 con 0x7f8b3c15cf80 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b24003520 con 0x7f8b3c15cf80 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4085261169 0 0) 0x7f8b24003520 con 0x7f8b3c15cf80 2024-04-15T15:16:50.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 msgr2=0x7f8b3c15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b3c15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.193 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 >> v2:172.21.15.96:3300/0 conn(0x7f8b3c15c660 msgr2=0x7f8b3c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 --2- 172.21.15.70:0/909732235 >> v2:172.21.15.96:3300/0 conn(0x7f8b3c15c660 0x7f8b3c06e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3c06f390 con 0x7f8b3c15cf80 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f8b340019f0 con 0x7f8b3c15cf80 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.185+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b3c15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/909732235 >> v2:172.21.15.96:3300/0 conn(0x7f8b3c15c660 0x7f8b3c06e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 >> v1:172.21.15.70:6789/0 conn(0x7f8b3c15cf80 legacy=0x7f8b3c0cfa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b240039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24003ef0 0x7f8b24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b24004d40 0x7f8b240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c06edd0 con 0x7f8b24003ef0 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c06f0b0 con 0x7f8b24004d40 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2400b9c0 con 0x7f8b3c0ccb20 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24003ef0 0x7f8b24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.194 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3b59e640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b24004d40 0x7f8b240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b240039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 -- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 msgr2=0x7f8b240039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b240039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 -- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b24004d40 msgr2=0x7f8b240051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b24004d40 0x7f8b240051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3a59c640 1 -- 172.21.15.70:0/909732235 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b24003520 con 0x7f8b24003ef0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.186+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b3c0ccb20 0x7f8b240039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24003ef0 0x7f8b24004800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009c90 tx=0x7f8b24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ab68657df829cf6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30039070 con 0x7f8b24003ef0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b3002c8f0 con 0x7f8b24003ef0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b39d9b640 1 -- 172.21.15.70:0/909732235 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b30009860 con 0x7f8b24003ef0 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 -- 172.21.15.70:0/909732235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24003ef0 msgr2=0x7f8b24004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 --2- 172.21.15.70:0/909732235 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b24003ef0 0x7f8b24004800 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009c90 tx=0x7f8b24005830 comp rx=0 tx=0).stop 2024-04-15T15:16:50.195 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b3b59e640 1 -- 172.21.15.70:0/909732235 reap_dead start 2024-04-15T15:16:50.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 -- 172.21.15.70:0/909732235 shutdown_connections 2024-04-15T15:16:50.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 -- 172.21.15.70:0/909732235 >> 172.21.15.70:0/909732235 conn(0x7f8b3c06b9e0 msgr2=0x7f8b3c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 -- 172.21.15.70:0/909732235 shutdown_connections 2024-04-15T15:16:50.196 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.187+0000 7f8b4111c640 1 -- 172.21.15.70:0/909732235 wait complete. 2024-04-15T15:16:50.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.189+0000 7f8b4111c640 1 Processor -- start 2024-04-15T15:16:50.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.189+0000 7f8b4111c640 1 -- start start 2024-04-15T15:16:50.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 0x7f8b3c146a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 0x7f8b3c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c0d0180 con 0x7f8b3c148920 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3c0d1110 con 0x7f8b3c1472c0 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8b3c0d0300 con 0x7f8b24003ef0 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49936/0 (socket says 172.21.15.70:49936) 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 -- 172.21.15.70:0/438737044 learned_addr learned my addr 172.21.15.70:0/438737044 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 0x7f8b3c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 0x7f8b3c146a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 -- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 msgr2=0x7f8b3c146a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 0x7f8b3c146a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 -- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 msgr2=0x7f8b3c0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 0x7f8b3c0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 -- 172.21.15.70:0/438737044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3c146fb0 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3a59c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 0x7f8b3c0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 0x7f8b3c146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b3b59e640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c069cc0 tx=0x7f8b34007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3400ac20 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b34018070 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3c147880 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b340133f0 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.190+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b3c147e40 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.192+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b3c06c970 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.192+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8b34013590 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.192+0000 7f8b237fe640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 0x7f8b2806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.192+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f8b3408c3a0 con 0x7f8b3c148920 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.192+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 0x7f8b2806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.200 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.193+0000 7f8b3ad9d640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 0x7f8b2806ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009c10 tx=0x7f8b2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.197+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8b3405a5c0 con 0x7f8b3c148920 2024-04-15T15:16:50.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.381+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8b3c072490 con 0x7f8b3c148920 2024-04-15T15:16:50.383 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.381+0000 7f8b237fe640 1 -- 172.21.15.70:0/438737044 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v39) ==== 80+0+0 (secure 0 0 0) 0x7f8b34059ec0 con 0x7f8b3c148920 2024-04-15T15:16:50.383 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 msgr2=0x7f8b2806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 0x7f8b2806ee40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c009c10 tx=0x7f8b2c008040 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 msgr2=0x7f8b3c0ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c069cc0 tx=0x7f8b34007bd0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 shutdown_connections 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8b2806c980 0x7f8b2806ee40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b3c148920 0x7f8b3c0ca940 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8b3c1472c0 0x7f8b3c0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 --2- 172.21.15.70:0/438737044 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8b24003ef0 0x7f8b3c146a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 >> 172.21.15.70:0/438737044 conn(0x7f8b3c06b9e0 msgr2=0x7f8b3c1556a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.383+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 shutdown_connections 2024-04-15T15:16:50.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.384+0000 7f8b4111c640 1 -- 172.21.15.70:0/438737044 wait complete. 2024-04-15T15:16:50.399 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:50.399 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:50.418 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:16:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.519+0000 7fbfac1b5640 1 Processor -- start 2024-04-15T15:16:50.521 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.519+0000 7fbfac1b5640 1 -- start start 2024-04-15T15:16:50.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfac1b5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfa415ecb0 0x7fbfa40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfac1b5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa40af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfac1b5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfa40b10d0 con 0x7fbfa40b4350 2024-04-15T15:16:50.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfac1b5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fbfa415f090 con 0x7fbfa415ecb0 2024-04-15T15:16:50.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfac1b5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa40cdf80 con 0x7fbfa40b1d30 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfa9f51640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fbfa415ecb0 0x7fbfa40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfa9750640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbfa40b4350 0x7fbfa40b1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48334/0 (socket says 172.21.15.70:48334) 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.520+0000 7fbfa9750640 1 -- 172.21.15.70:0/2252212153 learned_addr learned my addr 172.21.15.70:0/2252212153 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfaa752640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa40af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa9f51640 1 -- 172.21.15.70:0/2252212153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 msgr2=0x7fbfa40af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa9f51640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa40af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa9f51640 1 -- 172.21.15.70:0/2252212153 >> v1:172.21.15.70:6789/0 conn(0x7fbfa40b4350 legacy=0x7fbfa40b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa9f51640 1 -- 172.21.15.70:0/2252212153 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa40af920 con 0x7fbfa415ecb0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa8f4f640 1 -- 172.21.15.70:0/2252212153 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1230753035 0 0) 0x7fbfa40b10d0 con 0x7fbfa40b4350 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.521+0000 7fbfa9f51640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa415ecb0 0x7fbfa40b0b90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb20dd8de8e388c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfa8f4f640 1 -- 172.21.15.70:0/2252212153 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa0034070 con 0x7fbfa415ecb0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfa8f4f640 1 -- 172.21.15.70:0/2252212153 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfa0030610 con 0x7fbfa415ecb0 2024-04-15T15:16:50.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfa8f4f640 1 -- 172.21.15.70:0/2252212153 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa00309e0 con 0x7fbfa415ecb0 2024-04-15T15:16:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2252212153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa415ecb0 msgr2=0x7fbfa40b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.528 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa415ecb0 0x7fbfa40b0b90 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fbfa00097a0 tx=0x7fbfa002b960 comp rx=0 tx=0).stop 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2252212153 shutdown_connections 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa40af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2252212153 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa415ecb0 0x7fbfa40b0b90 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2252212153 >> 172.21.15.70:0/2252212153 conn(0x7fbfa40b6870 msgr2=0x7fbfa40b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2252212153 shutdown_connections 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.522+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2252212153 wait complete. 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.523+0000 7fbfac1b5640 1 Processor -- start 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.523+0000 7fbfac1b5640 1 -- start start 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa4156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 0x7fbfa4149e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa40b47c0 con 0x7fbfa40b4350 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa40afee0 con 0x7fbfa415ecb0 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfac1b5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fbfa4148540 con 0x7fbfa40b1d30 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfa9750640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfa9750640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49948/0 (socket says 172.21.15.70:49948) 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfa9750640 1 -- 172.21.15.70:0/2164499940 learned_addr learned my addr 172.21.15.70:0/2164499940 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.524+0000 7fbfaa752640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 0x7fbfa4149e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 -- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 msgr2=0x7fbfa4156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9f51640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa4156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa4156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 -- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 msgr2=0x7fbfa4149e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 0x7fbfa4149e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 -- 172.21.15.70:0/2164499940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfa4157240 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfaa752640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 0x7fbfa4149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfa9750640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400c4a0 tx=0x7fbf9400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94018070 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf940074e0 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfa4147f30 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf94004a20 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.525+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfa409b790 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.526+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbfa40689e0 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.526+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fbf9401b070 con 0x7fbfa40b4350 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.527+0000 7fbf9a7fc640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 0x7fbf8806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.527+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbf9408aff0 con 0x7fbfa40b4350 2024-04-15T15:16:50.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.527+0000 7fbfa9f51640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 0x7fbf8806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.527+0000 7fbfa9f51640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 0x7fbf8806ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.531+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fbf94059330 con 0x7fbfa40b4350 2024-04-15T15:16:50.708 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.706+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbfa406c7c0 con 0x7fbfa40b4350 2024-04-15T15:16:50.709 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.707+0000 7fbf9a7fc640 1 -- 172.21.15.70:0/2164499940 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16882 (secure 0 0 0) 0x7fbf94058c30 con 0x7fbfa40b4350 2024-04-15T15:16:50.710 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:50.710 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":39,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:16:26.878225+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:16:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 msgr2=0x7fbf8806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 0x7fbf8806ee40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0036000 tx=0x7fbfa003b040 comp rx=0 tx=0).stop 2024-04-15T15:16:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 msgr2=0x7fbfa40ca8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fbf9400c4a0 tx=0x7fbf9400c960 comp rx=0 tx=0).stop 2024-04-15T15:16:50.712 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 shutdown_connections 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fbfa415ecb0 0x7fbfa4149e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fbf8806c980 0x7fbf8806ee40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbfa40b4350 0x7fbfa40ca8c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 --2- 172.21.15.70:0/2164499940 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fbfa40b1d30 0x7fbfa4156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 >> 172.21.15.70:0/2164499940 conn(0x7fbfa40b6870 msgr2=0x7fbfa40b3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 shutdown_connections 2024-04-15T15:16:50.713 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.709+0000 7fbfac1b5640 1 -- 172.21.15.70:0/2164499940 wait complete. 2024-04-15T15:16:50.723 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:16:50.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:16:50.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.827+0000 7f5d9cf9d640 1 Processor -- start 2024-04-15T15:16:50.829 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.827+0000 7f5d9cf9d640 1 -- start start 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d9cf9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f5d980d0d40 0x7f5d980cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d9cf9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5d980d4e60 0x7f5d980d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d9cf9d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d980d1660 con 0x7f5d980d2cc0 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d9cf9d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f5d980d1a00 con 0x7f5d980d4e60 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d9cf9d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f5d980d1f90 con 0x7f5d980d0d40 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d96ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f5d980d4e60 0x7f5d980d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d97fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5d980d2cc0 0x7f5d980b3d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48346/0 (socket says 172.21.15.70:48346) 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.828+0000 7f5d97fff640 1 -- 172.21.15.70:0/3935970045 learned_addr learned my addr 172.21.15.70:0/3935970045 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:50.831 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d977fe640 1 --2- 172.21.15.70:0/3935970045 >> v2:172.21.15.96:3300/0 conn(0x7f5d980d0d40 0x7f5d980cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d96ffd640 1 -- 172.21.15.70:0/3935970045 >> v2:172.21.15.96:3300/0 conn(0x7f5d980d0d40 msgr2=0x7f5d980cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:50.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d96ffd640 1 --2- 172.21.15.70:0/3935970045 >> v2:172.21.15.96:3300/0 conn(0x7f5d980d0d40 0x7f5d980cfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d96ffd640 1 -- 172.21.15.70:0/3935970045 >> v1:172.21.15.70:6789/0 conn(0x7f5d980d2cc0 legacy=0x7f5d980b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.832 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d96ffd640 1 -- 172.21.15.70:0/3935970045 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d980d02e0 con 0x7f5d980d4e60 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.829+0000 7f5d96ffd640 1 --2- 172.21.15.70:0/3935970045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d4e60 0x7f5d980d1120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5d980d1f40 tx=0x7f5d8c02b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f1faa739363c4e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d967fc640 1 -- 172.21.15.70:0/3935970045 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c038070 con 0x7f5d980d4e60 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d967fc640 1 -- 172.21.15.70:0/3935970045 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d8c031700 con 0x7f5d980d4e60 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d967fc640 1 -- 172.21.15.70:0/3935970045 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d8c031ad0 con 0x7f5d980d4e60 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/3935970045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d4e60 msgr2=0x7f5d980d1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/3935970045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d4e60 0x7f5d980d1120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f5d980d1f40 tx=0x7f5d8c02b7a0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/3935970045 shutdown_connections 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/3935970045 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d4e60 0x7f5d980d1120 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/3935970045 >> v2:172.21.15.96:3300/0 conn(0x7f5d980d0d40 0x7f5d980cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/3935970045 >> 172.21.15.70:0/3935970045 conn(0x7f5d9808bf60 msgr2=0x7f5d980d43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/3935970045 shutdown_connections 2024-04-15T15:16:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.830+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/3935970045 wait complete. 2024-04-15T15:16:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.831+0000 7f5d9cf9d640 1 Processor -- start 2024-04-15T15:16:50.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.831+0000 7f5d9cf9d640 1 -- start start 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d980d2cc0 0x7f5d9814a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d980d4e60 0x7f5d981e80d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5d980d27a0 con 0x7f5d980d0d40 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f5d980d24c0 con 0x7f5d980d2cc0 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d9cf9d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f5d9815fcc0 con 0x7f5d980d4e60 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d977fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d977fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49972/0 (socket says 172.21.15.70:49972) 2024-04-15T15:16:50.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.832+0000 7f5d977fe640 1 -- 172.21.15.70:0/4155296184 learned_addr learned my addr 172.21.15.70:0/4155296184 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 -- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d980d4e60 msgr2=0x7f5d981e80d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d97fff640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d980d4e60 0x7f5d981e80d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d96ffd640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d980d2cc0 0x7f5d9814a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d980d4e60 0x7f5d981e80d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 -- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d980d2cc0 msgr2=0x7f5d9814a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d980d2cc0 0x7f5d9814a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 -- 172.21.15.70:0/4155296184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d980b8260 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d977fe640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5d88007e70 tx=0x7f5d8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88014620 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d88019070 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88014a00 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d980b8540 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.833+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d980b8b00 con 0x7f5d980d0d40 2024-04-15T15:16:50.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.834+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9804fd50 con 0x7f5d980d0d40 2024-04-15T15:16:50.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.834+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5d88014ba0 con 0x7f5d980d0d40 2024-04-15T15:16:50.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.835+0000 7f5d7ffff640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 0x7f5d8406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:50.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.835+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f5d8808c5c0 con 0x7f5d980d0d40 2024-04-15T15:16:50.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.835+0000 7f5d96ffd640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 0x7f5d8406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:50.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.835+0000 7f5d96ffd640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 0x7f5d8406ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c035000 tx=0x7f5d8c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:50.841 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:50.839+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f5d8805a870 con 0x7f5d980d0d40 2024-04-15T15:16:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.013+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5d980b56b0 con 0x7f5d980d0d40 2024-04-15T15:16:51.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.013+0000 7f5d7ffff640 1 -- 172.21.15.70:0/4155296184 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v39) ==== 93+0+10 (secure 0 0 0) 0x7f5d8805a170 con 0x7f5d980d0d40 2024-04-15T15:16:51.015 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 msgr2=0x7f5d8406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 0x7f5d8406ee40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5d8c035000 tx=0x7f5d8c008030 comp rx=0 tx=0).stop 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 msgr2=0x7f5d981481e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f5d88007e70 tx=0x7f5d8800e4b0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 shutdown_connections 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f5d980d4e60 0x7f5d981e80d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f5d980d2cc0 0x7f5d9814a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f5d8406c980 0x7f5d8406ee40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 --2- 172.21.15.70:0/4155296184 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5d980d0d40 0x7f5d981481e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 >> 172.21.15.70:0/4155296184 conn(0x7f5d9808bf60 msgr2=0x7f5d980b0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 shutdown_connections 2024-04-15T15:16:51.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.015+0000 7f5d9cf9d640 1 -- 172.21.15.70:0/4155296184 wait complete. 2024-04-15T15:16:51.028 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:16:51.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.132+0000 7f4a49fe2640 1 Processor -- start 2024-04-15T15:16:51.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.132+0000 7f4a49fe2640 1 -- start start 2024-04-15T15:16:51.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a49fe2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440b3160 0x7f4a4415b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a49fe2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a440ae280 0x7f4a440ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a49fe2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a4415cb50 con 0x7f4a440d0690 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a49fe2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f4a4415bfd0 con 0x7f4a440ae280 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a49fe2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4415c2b0 con 0x7f4a440b3160 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a440ae280 0x7f4a440ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4a440ae280 0x7f4a440ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49976/0 (socket says 172.21.15.70:49976) 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 -- 172.21.15.70:0/1098993764 learned_addr learned my addr 172.21.15.70:0/1098993764 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a42ffd640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440b3160 0x7f4a4415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 -- 172.21.15.70:0/1098993764 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440b3160 msgr2=0x7f4a4415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440b3160 0x7f4a4415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 -- 172.21.15.70:0/1098993764 >> v1:172.21.15.96:6789/0 conn(0x7f4a440d0690 legacy=0x7f4a440d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.133+0000 7f4a43fff640 1 -- 172.21.15.70:0/1098993764 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4415c590 con 0x7f4a440ae280 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.134+0000 7f4a43fff640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440ae280 0x7f4a440ae6e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009870 tx=0x7f4a3c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8127f1d5464e3698 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.134+0000 7f4a427fc640 1 -- 172.21.15.70:0/1098993764 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c034070 con 0x7f4a440ae280 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.134+0000 7f4a427fc640 1 -- 172.21.15.70:0/1098993764 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a3c030840 con 0x7f4a440ae280 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.134+0000 7f4a427fc640 1 -- 172.21.15.70:0/1098993764 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c030c10 con 0x7f4a440ae280 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 -- 172.21.15.70:0/1098993764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440ae280 msgr2=0x7f4a440ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440ae280 0x7f4a440ae6e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009870 tx=0x7f4a3c02b980 comp rx=0 tx=0).stop 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 -- 172.21.15.70:0/1098993764 shutdown_connections 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440ae280 0x7f4a440ae6e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/1098993764 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440b3160 0x7f4a4415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 -- 172.21.15.70:0/1098993764 >> 172.21.15.70:0/1098993764 conn(0x7f4a4406be00 msgr2=0x7f4a4406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 -- 172.21.15.70:0/1098993764 shutdown_connections 2024-04-15T15:16:51.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.135+0000 7f4a49fe2640 1 -- 172.21.15.70:0/1098993764 wait complete. 2024-04-15T15:16:51.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.136+0000 7f4a49fe2640 1 Processor -- start 2024-04-15T15:16:51.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.136+0000 7f4a49fe2640 1 -- start start 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 0x7f4a440cf540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 0x7f4a441f8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a4415bbd0 con 0x7f4a440b3160 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4a440b4fb0 con 0x7f4a440ae280 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a49fe2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4a4415bd50 con 0x7f4a440d0690 2024-04-15T15:16:51.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49992/0 (socket says 172.21.15.70:49992) 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 -- 172.21.15.70:0/2823260503 learned_addr learned my addr 172.21.15.70:0/2823260503 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a43fff640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 0x7f4a441f8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a437fe640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 0x7f4a440cf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 -- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 msgr2=0x7f4a441f8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 0x7f4a441f8640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 -- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 msgr2=0x7f4a440cf540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 0x7f4a440cf540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a42ffd640 1 -- 172.21.15.70:0/2823260503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a440cfa80 con 0x7f4a440b3160 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a437fe640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 0x7f4a440cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:51.144 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.137+0000 7f4a43fff640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 0x7f4a441f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a42ffd640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800d5d0 tx=0x7f4a3800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a380125f0 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a38019070 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4415ebb0 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a380129f0 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.138+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a4415f170 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.139+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a440689e0 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.139+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4a38012b90 con 0x7f4a440b3160 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.140+0000 7f4a2bfff640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 0x7f4a3006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.140+0000 7f4a437fe640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 0x7f4a3006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.145 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.140+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f4a3808c120 con 0x7f4a440b3160 2024-04-15T15:16:51.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.140+0000 7f4a437fe640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 0x7f4a3006ee40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4a440cf4d0 tx=0x7f4a34009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.144+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4a3805a460 con 0x7f4a440b3160 2024-04-15T15:16:51.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.318+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4a4415f340 con 0x7f4a440b3160 2024-04-15T15:16:51.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.319+0000 7f4a2bfff640 1 -- 172.21.15.70:0/2823260503 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v39) ==== 104+0+11 (secure 0 0 0) 0x7f4a4415f340 con 0x7f4a440b3160 2024-04-15T15:16:51.321 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 msgr2=0x7f4a3006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 0x7f4a3006ee40 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4a440cf4d0 tx=0x7f4a34009040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 msgr2=0x7f4a441f8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800d5d0 tx=0x7f4a3800da90 comp rx=0 tx=0).stop 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 shutdown_connections 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4a440d0690 0x7f4a441f8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4a3006c980 0x7f4a3006ee40 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a440b3160 0x7f4a441f8100 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 --2- 172.21.15.70:0/2823260503 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4a440ae280 0x7f4a440cf540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 >> 172.21.15.70:0/2823260503 conn(0x7f4a4406be00 msgr2=0x7f4a44074de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 shutdown_connections 2024-04-15T15:16:51.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.321+0000 7f4a49fe2640 1 -- 172.21.15.70:0/2823260503 wait complete. 2024-04-15T15:16:51.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:16:51.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.443+0000 7efcd3d1c640 1 Processor -- start 2024-04-15T15:16:51.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 -- start start 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc15e700 0x7efccc0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efccc0b47a0 con 0x7efccc0d2360 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efccc0b53b0 con 0x7efccc15e700 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd3d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efccc0b4a80 con 0x7efccc0d3270 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.444+0000 7efcd1ab8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc15e700 0x7efccc0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efccc15e700 0x7efccc0b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50008/0 (socket says 172.21.15.70:50008) 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 -- 172.21.15.70:0/3975864019 learned_addr learned my addr 172.21.15.70:0/3975864019 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd22b9640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 -- 172.21.15.70:0/3975864019 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 msgr2=0x7efccc0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0cf370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 -- 172.21.15.70:0/3975864019 >> v1:172.21.15.96:6789/0 conn(0x7efccc0d2360 legacy=0x7efccc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.445+0000 7efcd1ab8640 1 -- 172.21.15.70:0/3975864019 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc0b4d60 con 0x7efccc15e700 2024-04-15T15:16:51.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd1ab8640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b5840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009ea0 tx=0x7efcbc02bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e654007210863559 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd0ab6640 1 -- 172.21.15.70:0/3975864019 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc02cab0 con 0x7efccc15e700 2024-04-15T15:16:51.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd0ab6640 1 -- 172.21.15.70:0/3975864019 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcbc02cc50 con 0x7efccc15e700 2024-04-15T15:16:51.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd0ab6640 1 -- 172.21.15.70:0/3975864019 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcbc0343f0 con 0x7efccc15e700 2024-04-15T15:16:51.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 -- 172.21.15.70:0/3975864019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 msgr2=0x7efccc0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b5840 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7efcbc009ea0 tx=0x7efcbc02bf10 comp rx=0 tx=0).stop 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 -- 172.21.15.70:0/3975864019 shutdown_connections 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/3975864019 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b5840 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 -- 172.21.15.70:0/3975864019 >> 172.21.15.70:0/3975864019 conn(0x7efccc08bf60 msgr2=0x7efccc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 -- 172.21.15.70:0/3975864019 shutdown_connections 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.446+0000 7efcd3d1c640 1 -- 172.21.15.70:0/3975864019 wait complete. 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 Processor -- start 2024-04-15T15:16:51.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 -- start start 2024-04-15T15:16:51.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efccc0d2360 0x7efccc156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0b9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.452 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efccc0b5d80 con 0x7efccc15e700 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efccc0b4450 con 0x7efccc0d2360 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd3d1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efccc0d1d40 con 0x7efccc0d3270 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd22b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd22b9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50014/0 (socket says 172.21.15.70:50014) 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.448+0000 7efcd22b9640 1 -- 172.21.15.70:0/2428315601 learned_addr learned my addr 172.21.15.70:0/2428315601 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd1ab8640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efccc0d2360 0x7efccc156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 -- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 msgr2=0x7efccc0b9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd12b7640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0b9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0b9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 -- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efccc0d2360 msgr2=0x7efccc156e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efccc0d2360 0x7efccc156e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 -- 172.21.15.70:0/2428315601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efccc1573a0 con 0x7efccc15e700 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd22b9640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c7f0 tx=0x7efcc400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4015070 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcc4004220 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efccc0b3e00 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc4007620 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.449+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efccc0cc680 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.450+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efccc0690f0 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.450+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7efcc40077c0 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.451+0000 7efcca7fc640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 0x7efcb806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.451+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7efcc408bc60 con 0x7efccc15e700 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.451+0000 7efcd1ab8640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 0x7efcb806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.451+0000 7efcd1ab8640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 0x7efcb806ee70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efcbc035000 tx=0x7efcbc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.455+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efccc0690f0 con 0x7efccc15e700 2024-04-15T15:16:51.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.631+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efccc0689e0 con 0x7efccc15e700 2024-04-15T15:16:51.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.632+0000 7efcca7fc640 1 -- 172.21.15.70:0/2428315601 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) ==== 100+0+11 (secure 0 0 0) 0x7efcc4059de0 con 0x7efccc15e700 2024-04-15T15:16:51.634 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.633+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 msgr2=0x7efcb806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.633+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 0x7efcb806ee70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7efcbc035000 tx=0x7efcbc030000 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.633+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 msgr2=0x7efccc0b9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.633+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7efcc400c7f0 tx=0x7efcc400ccb0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 shutdown_connections 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efcb806c9b0 0x7efcb806ee70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efccc15e700 0x7efccc0b9cd0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efccc0d3270 0x7efccc0b9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 --2- 172.21.15.70:0/2428315601 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efccc0d2360 0x7efccc156e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.636 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 >> 172.21.15.70:0/2428315601 conn(0x7efccc08bf60 msgr2=0x7efccc0c7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 shutdown_connections 2024-04-15T15:16:51.637 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.634+0000 7efcd3d1c640 1 -- 172.21.15.70:0/2428315601 wait complete. 2024-04-15T15:16:51.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:16:51.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.749+0000 7f3a23e0d640 1 Processor -- start 2024-04-15T15:16:51.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.749+0000 7f3a23e0d640 1 -- start start 2024-04-15T15:16:51.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a23e0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a1c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a23e0d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f3a1c0ae280 0x7f3a1c0ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a23e0d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a1c15cb50 con 0x7f3a1c0d0690 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a23e0d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f3a1c15bfd0 con 0x7f3a1c0ae280 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a23e0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c15c2b0 con 0x7f3a1c0b3160 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a21ba9640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3a1c0d0690 0x7f3a1c0d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48358/0 (socket says 172.21.15.70:48358) 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a21ba9640 1 -- 172.21.15.70:0/3151744157 learned_addr learned my addr 172.21.15.70:0/3151744157 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a223aa640 1 --2- 172.21.15.70:0/3151744157 >> v2:172.21.15.96:3300/0 conn(0x7f3a1c0ae280 0x7f3a1c0ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561415933 0 0) 0x7f3a1c15cb50 con 0x7f3a1c0d0690 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a213a8640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a1c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.750+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a08003520 con 0x7f3a1c0d0690 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1770208813 0 0) 0x7f3a08003520 con 0x7f3a1c0d0690 2024-04-15T15:16:51.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 msgr2=0x7f3a1c15b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a1c15b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 >> v2:172.21.15.96:3300/0 conn(0x7f3a1c0ae280 msgr2=0x7f3a1c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 --2- 172.21.15.70:0/3151744157 >> v2:172.21.15.96:3300/0 conn(0x7f3a1c0ae280 0x7f3a1c0ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c15c590 con 0x7f3a1c0d0690 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f3a0c001670 con 0x7f3a1c0d0690 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 >> v1:172.21.15.70:6789/0 conn(0x7f3a1c0d0690 legacy=0x7f3a1c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a080039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a08003f30 0x7f3a08004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a080051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c15bfd0 con 0x7f3a08004d40 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c15c2b0 con 0x7f3a08003f30 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a0800b9c0 con 0x7f3a1c0b3160 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a080051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a21ba9640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a080039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a213a8640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a08003f30 0x7f3a08004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 -- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 msgr2=0x7f3a080039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c0b3160 0x7f3a080039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 -- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a08003f30 msgr2=0x7f3a08004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a08003f30 0x7f3a08004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.751+0000 7f3a223aa640 1 -- 172.21.15.70:0/3151744157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a08003520 con 0x7f3a08004d40 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a223aa640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a080051c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009fc0 tx=0x7f3a08005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14c206fb49fc25 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14039070 con 0x7f3a08004d40 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a1402cc40 con 0x7f3a08004d40 2024-04-15T15:16:51.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a20ba7640 1 -- 172.21.15.70:0/3151744157 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14009c10 con 0x7f3a08004d40 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a23e0d640 1 -- 172.21.15.70:0/3151744157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 msgr2=0x7f3a080051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/3151744157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a080051c0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009fc0 tx=0x7f3a08005830 comp rx=0 tx=0).stop 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a223aa640 1 -- 172.21.15.70:0/3151744157 reap_dead start 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a23e0d640 1 -- 172.21.15.70:0/3151744157 shutdown_connections 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a23e0d640 1 -- 172.21.15.70:0/3151744157 >> 172.21.15.70:0/3151744157 conn(0x7f3a1c06be00 msgr2=0x7f3a1c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.752+0000 7f3a23e0d640 1 -- 172.21.15.70:0/3151744157 shutdown_connections 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.753+0000 7f3a23e0d640 1 -- 172.21.15.70:0/3151744157 wait complete. 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 Processor -- start 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 -- start start 2024-04-15T15:16:51.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a1c0b1b30 0x7f3a1c0b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c148c20 0x7f3a1c1490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c15b690 con 0x7f3a08004d40 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c0b4fb0 con 0x7f3a1c0b1b30 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.754+0000 7f3a23e0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c15b810 con 0x7f3a1c148c20 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50040/0 (socket says 172.21.15.70:50040) 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 -- 172.21.15.70:0/1964174111 learned_addr learned my addr 172.21.15.70:0/1964174111 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a213a8640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a1c0b1b30 0x7f3a1c0b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a223aa640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c148c20 0x7f3a1c1490d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 -- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c148c20 msgr2=0x7f3a1c1490d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c148c20 0x7f3a1c1490d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 -- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a1c0b1b30 msgr2=0x7f3a1c0b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a1c0b1b30 0x7f3a1c0b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 -- 172.21.15.70:0/1964174111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c149990 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a21ba9640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c007370 tx=0x7f3a0c007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c01a070 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00fa60 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c146ae0 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00fe50 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.755+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c1470a0 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.756+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1c0689e0 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.757+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3a0c01d360 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.757+0000 7f3a127fc640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 0x7f3a0006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.757+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f3a0c08b7e0 con 0x7f3a08004d40 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.757+0000 7f3a213a8640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 0x7f3a0006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:51.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.758+0000 7f3a213a8640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 0x7f3a0006ee70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0cdd60 tx=0x7f3a18008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:51.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.761+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a0c059a70 con 0x7f3a08004d40 2024-04-15T15:16:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.948+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3a1c0739e0 con 0x7f3a08004d40 2024-04-15T15:16:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.948+0000 7f3a127fc640 1 -- 172.21.15.70:0/1964174111 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) ==== 80+0+0 (secure 0 0 0) 0x7f3a0c059370 con 0x7f3a08004d40 2024-04-15T15:16:51.950 INFO:teuthology.orchestra.run.smithi070.stderr:listed 0 entries 2024-04-15T15:16:51.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 msgr2=0x7f3a0006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 0x7f3a0006ee70 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0cdd60 tx=0x7f3a18008040 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 msgr2=0x7f3a1c1f75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c007370 tx=0x7f3a0c007830 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 shutdown_connections 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a1c148c20 0x7f3a1c1490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a1c0b1b30 0x7f3a1c0b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a0006c9b0 0x7f3a0006ee70 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 --2- 172.21.15.70:0/1964174111 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a08004d40 0x7f3a1c1f75d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 >> 172.21.15.70:0/1964174111 conn(0x7f3a1c06be00 msgr2=0x7f3a1c0d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 shutdown_connections 2024-04-15T15:16:51.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:51.950+0000 7f3a23e0d640 1 -- 172.21.15.70:0/1964174111 wait complete. 2024-04-15T15:16:51.963 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-15T15:16:51.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:51.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:51.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45162]: fuse finished with error 0 and tester_r 0 2024-04-15T15:16:51.997 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:16:52.070 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:16:52.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.070 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:16:52.088 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:16:52.088 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:52.105 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19879 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:16:52.106 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.2 is not mounted at ubuntu@smithi070.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:52.106 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:16:52.106 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.106 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:16:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:16:52.162 INFO:teuthology.orchestra.run.smithi070.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-15T15:16:52.162 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:16:52.162 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:16:52.180 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056369,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=553 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248840k,nr_inodes=812210,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:16:52.180 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.3 is not mounted at ubuntu@smithi096.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:52.180 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:16:52.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.181 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:16:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:16:52.238 INFO:teuthology.orchestra.run.smithi096.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-15T15:16:52.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:52.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.344+0000 7f7889baf640 1 Processor -- start 2024-04-15T15:16:52.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.344+0000 7f7889baf640 1 -- start start 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7889baf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7884156df0 0x7f78840cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7889baf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f788415d4b0 con 0x7f7884160e30 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7889baf640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f788415d790 con 0x7f78841609d0 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7889baf640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f788415d250 con 0x7f7884156df0 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f78837fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f7884156df0 0x7f78840cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7883fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7884160e30 0x7f78840b5f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48368/0 (socket says 172.21.15.70:48368) 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f7883fff640 1 -- 172.21.15.70:0/3972442870 learned_addr learned my addr 172.21.15.70:0/3972442870 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f78837fe640 1 -- 172.21.15.70:0/3972442870 >> v1:172.21.15.96:6789/0 conn(0x7f78841609d0 legacy=0x7f788415cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f78837fe640 1 -- 172.21.15.70:0/3972442870 >> v1:172.21.15.70:6789/0 conn(0x7f7884160e30 legacy=0x7f78840b5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f78837fe640 1 -- 172.21.15.70:0/3972442870 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f788415da70 con 0x7f7884156df0 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.345+0000 7f78827fc640 1 -- 172.21.15.70:0/3972442870 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 894444168 0 0) 0x7f788415d4b0 con 0x7f7884160e30 2024-04-15T15:16:52.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.346+0000 7f78837fe640 1 --2- 172.21.15.70:0/3972442870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884156df0 0x7f78840cf320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d90 tx=0x7f787402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61854b0765edf14c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.346+0000 7f78827fc640 1 -- 172.21.15.70:0/3972442870 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874039070 con 0x7f7884156df0 2024-04-15T15:16:52.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.346+0000 7f78827fc640 1 -- 172.21.15.70:0/3972442870 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787402cbd0 con 0x7f7884156df0 2024-04-15T15:16:52.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.346+0000 7f78827fc640 1 -- 172.21.15.70:0/3972442870 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7874034440 con 0x7f7884156df0 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 -- 172.21.15.70:0/3972442870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884156df0 msgr2=0x7f78840cf320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 --2- 172.21.15.70:0/3972442870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884156df0 0x7f78840cf320 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f7874009d90 tx=0x7f787402be20 comp rx=0 tx=0).stop 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 -- 172.21.15.70:0/3972442870 shutdown_connections 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 --2- 172.21.15.70:0/3972442870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884156df0 0x7f78840cf320 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 -- 172.21.15.70:0/3972442870 >> 172.21.15.70:0/3972442870 conn(0x7f7884073380 msgr2=0x7f788406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 -- 172.21.15.70:0/3972442870 shutdown_connections 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.347+0000 7f7889baf640 1 -- 172.21.15.70:0/3972442870 wait complete. 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.348+0000 7f7889baf640 1 Processor -- start 2024-04-15T15:16:52.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.348+0000 7f7889baf640 1 -- start start 2024-04-15T15:16:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78841609d0 0x7f7884152640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 0x7f78841538f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f78840b66a0 con 0x7f7884153e30 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f78841604b0 con 0x7f7884160e30 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7889baf640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f78840b6820 con 0x7f78841609d0 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:50056/0 (socket says 172.21.15.70:50056) 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 -- 172.21.15.70:0/1380552633 learned_addr learned my addr 172.21.15.70:0/1380552633 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7882ffd640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 0x7f78841538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f78837fe640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78841609d0 0x7f7884152640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 -- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78841609d0 msgr2=0x7f7884152640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78841609d0 0x7f7884152640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 -- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 msgr2=0x7f78841538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 0x7f78841538f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 -- 172.21.15.70:0/1380552633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78840b5790 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7882ffd640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 0x7f78841538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f7883fff640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c440 tx=0x7f787c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.349+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c0074e0 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.350+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f787c0047e0 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.350+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7884152b80 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.350+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f787c004bd0 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.350+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78840b4c40 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.350+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78840689e0 con 0x7f7884153e30 2024-04-15T15:16:52.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.351+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f787c011070 con 0x7f7884153e30 2024-04-15T15:16:52.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.351+0000 7f786bfff640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 0x7f787006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:52.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.352+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6130+0+0 (secure 0 0 0) 0x7f787c08b7d0 con 0x7f7884153e30 2024-04-15T15:16:52.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.352+0000 7f78837fe640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 0x7f787006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:52.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.352+0000 7f78837fe640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 0x7f787006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7874036000 tx=0x7f7874030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:52.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.356+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f7884152b80 con 0x7f7884153e30 2024-04-15T15:16:52.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.570+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f7884067d90 con 0x7f7884153e30 2024-04-15T15:16:52.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.570+0000 7f786bfff640 1 -- 172.21.15.70:0/1380552633 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f78840b5790 con 0x7f7884153e30 2024-04-15T15:16:52.572 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 msgr2=0x7f787006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 0x7f787006ee40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7874036000 tx=0x7f7874030000 comp rx=0 tx=0).stop 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 msgr2=0x7f78841e8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f787c00c440 tx=0x7f787c00c900 comp rx=0 tx=0).stop 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 shutdown_connections 2024-04-15T15:16:52.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f787006c980 0x7f787006ee40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7884153e30 0x7f78841e8160 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7884160e30 0x7f78841538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 --2- 172.21.15.70:0/1380552633 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f78841609d0 0x7f7884152640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 >> 172.21.15.70:0/1380552633 conn(0x7f7884073380 msgr2=0x7f788406c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 shutdown_connections 2024-04-15T15:16:52.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52.572+0000 7f7889baf640 1 -- 172.21.15.70:0/1380552633 wait complete. 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:52.585 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:52.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.586 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:16:52.602 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:52.602 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:52.602 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:52.602 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:52.603 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42638sec preferred_lft 42638sec 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c488:39ff:fe35:ded1/64 scope link 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 82:8a:03:3a:60:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::808a:3ff:fe3a:6003/64 scope link 2024-04-15T15:16:52.604 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.605 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:52.605 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:52.605 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::6898:acff:fe89:2456/64 scope link 2024-04-15T15:16:52.605 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:52.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.605 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:52.669 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:16:52.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.671 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:16:52.728 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:16:52.728 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:52.728 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:52.729 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:52.729 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:52.729 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:16:52.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:52.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.868 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:16:52.884 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:52.884 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:52.884 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:52.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:52.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.0 up 2024-04-15T15:16:52.953 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-15T15:16:52.953 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:52.953 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:53.009 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-15T15:16:53.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:53.010 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:53.065 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:53.133 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:53.190 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:53.191 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:53.192 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:53.193 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:53.193 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:53.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:53.193 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:53.246 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:53.246 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb) 2024-04-15T15:16:53.289 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:53.317 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:53.374 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:53.375 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:53.376 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:53.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:53.377 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:53.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:2024-04-15T15:16:53.375+0000 7f9fcbe38480 -1 init, newargv = 0x55c458a83f50 newargc=15 2024-04-15T15:16:53.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[55292]: starting ceph client 2024-04-15T15:16:53.386 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[55292]: starting fuse 2024-04-15T15:16:53.393 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:16:53.393 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-15T15:16:53.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:53.393 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:16:53.394 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.0")) 2024-04-15T15:16:53.395 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:16:53.515 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.0.55292.asok 2024-04-15T15:16:53.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:53.525 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55292.asok 2024-04-15T15:16:53.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:53.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55292.asok status 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "0", 2024-04-15T15:16:53.663 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "55292", 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb" 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5940, 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 5940 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:53.664 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1475250011 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:53.665 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 1475250011 2024-04-15T15:16:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.5940 192.168.0.1:0/1475250011", 2024-04-15T15:16:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.1:0/1475250011", 2024-04-15T15:16:53.666 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 5, 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 15, 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 41, 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:16:53.667 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:53.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:53.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.774+0000 7ff97350e640 1 Processor -- start 2024-04-15T15:16:53.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.774+0000 7ff97350e640 1 -- start start 2024-04-15T15:16:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff97350e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff96c09b0e0 0x7ff96c0d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff97350e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c15e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff97350e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff96c06cff0 con 0x7ff96c0b1590 2024-04-15T15:16:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff97350e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff96c06dc00 con 0x7ff96c09b0e0 2024-04-15T15:16:53.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff97350e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff96c06d2d0 con 0x7ff96c06caf0 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff970aa9640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff96c09b0e0 0x7ff96c0d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff971aab640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c15e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff9712aa640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7ff96c0b1590 0x7ff96c09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:45192/0 (socket says 172.21.15.70:45192) 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff9712aa640 1 -- 172.21.15.70:0/454234882 learned_addr learned my addr 172.21.15.70:0/454234882 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff970aa9640 1 -- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 msgr2=0x7ff96c15e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff970aa9640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c15e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff970aa9640 1 -- 172.21.15.70:0/454234882 >> v1:172.21.15.96:6789/0 conn(0x7ff96c0b1590 legacy=0x7ff96c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff963fff640 1 -- 172.21.15.70:0/454234882 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3276048779 0 0) 0x7ff96c06cff0 con 0x7ff96c0b1590 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff970aa9640 1 -- 172.21.15.70:0/454234882 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96c06d5b0 con 0x7ff96c09b0e0 2024-04-15T15:16:53.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.775+0000 7ff971aab640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:53.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.776+0000 7ff970aa9640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c09b0e0 0x7ff96c0d16b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0097a0 tx=0x7ff95c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efb3a8d13955876c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.776+0000 7ff963fff640 1 -- 172.21.15.70:0/454234882 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c035020 con 0x7ff96c09b0e0 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.776+0000 7ff963fff640 1 -- 172.21.15.70:0/454234882 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c031830 con 0x7ff96c09b0e0 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.776+0000 7ff963fff640 1 -- 172.21.15.70:0/454234882 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c031c00 con 0x7ff96c09b0e0 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 -- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c09b0e0 msgr2=0x7ff96c0d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c09b0e0 0x7ff96c0d16b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff95c0097a0 tx=0x7ff95c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 -- 172.21.15.70:0/454234882 shutdown_connections 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c15e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 --2- 172.21.15.70:0/454234882 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c09b0e0 0x7ff96c0d16b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 -- 172.21.15.70:0/454234882 >> 172.21.15.70:0/454234882 conn(0x7ff96c0b3e90 msgr2=0x7ff96c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 -- 172.21.15.70:0/454234882 shutdown_connections 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.777+0000 7ff97350e640 1 -- 172.21.15.70:0/454234882 wait complete. 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.778+0000 7ff97350e640 1 Processor -- start 2024-04-15T15:16:53.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.778+0000 7ff97350e640 1 -- start start 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c1f8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 0x7ff96c0b72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c0d0b20 con 0x7ff96c09b0e0 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff96c0d09a0 con 0x7ff96c0b1590 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff97350e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff96c0b1970 con 0x7ff96c06caf0 2024-04-15T15:16:53.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46678/0 (socket says 172.21.15.70:46678) 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 -- 172.21.15.70:0/1835287319 learned_addr learned my addr 172.21.15.70:0/1835287319 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff971aab640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 0x7ff96c0b72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff9712aa640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c1f8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 -- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 msgr2=0x7ff96c1f8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c1f8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 -- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 msgr2=0x7ff96c0b72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 0x7ff96c0b72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff970aa9640 1 -- 172.21.15.70:0/1835287319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff96c0b6240 con 0x7ff96c09b0e0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.779+0000 7ff971aab640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 0x7ff96c0b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff970aa9640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff95c000c00 tx=0x7ff95c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c035070 con 0x7ff96c09b0e0 2024-04-15T15:16:53.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff95c004230 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff96c0b6520 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff95c03a630 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.780+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff96c0b6dc0 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.781+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff96c0689e0 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.781+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff95c038020 con 0x7ff96c09b0e0 2024-04-15T15:16:53.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.782+0000 7ff961ffb640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 0x7ff95006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.782+0000 7ff9712aa640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 0x7ff95006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.782+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7ff95c0ada20 con 0x7ff96c09b0e0 2024-04-15T15:16:53.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.782+0000 7ff9712aa640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 0x7ff95006ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff968007fd0 tx=0x7ff968007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:53.788 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:53.786+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff95c040070 con 0x7ff96c09b0e0 2024-04-15T15:16:54.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.006+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ff96c0b4a80 con 0x7ff96c09b0e0 2024-04-15T15:16:54.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.006+0000 7ff961ffb640 1 -- 172.21.15.70:0/1835287319 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7ff95c07b4d0 con 0x7ff96c09b0e0 2024-04-15T15:16:54.008 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 msgr2=0x7ff95006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 0x7ff95006ee40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff968007fd0 tx=0x7ff968007de0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 msgr2=0x7ff96c1e70f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff95c000c00 tx=0x7ff95c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 shutdown_connections 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff96c0b1590 0x7ff96c0b72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff95006c980 0x7ff95006ee40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff96c09b0e0 0x7ff96c1e70f0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 --2- 172.21.15.70:0/1835287319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff96c06caf0 0x7ff96c1f8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.008+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 >> 172.21.15.70:0/1835287319 conn(0x7ff96c0b3e90 msgr2=0x7ff96c09a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.009+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 shutdown_connections 2024-04-15T15:16:54.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:54.009+0000 7ff97350e640 1 -- 172.21.15.70:0/1835287319 wait complete. 2024-04-15T15:16:54.021 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:54.021 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:54.021 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-15T15:16:54.021 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-15T15:16:54.022 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:54.022 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:54.022 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 2024-04-15T15:16:54.022 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:54.022 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:54.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.022 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-15T15:16:54.042 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:54.042 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:54.042 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:54.042 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42611sec preferred_lft 42611sec 2024-04-15T15:16:54.043 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::c27:e4ff:fe37:c79c/64 scope link 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether f6:ed:c4:10:c9:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::f4ed:c4ff:fe10:c9d9/64 scope link 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::d096:56ff:fe49:5376/64 scope link 2024-04-15T15:16:54.044 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:54.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.045 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:54.117 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:16:54.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.119 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-15T15:16:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-15T15:16:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:54.178 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:54.178 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:54.179 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:16:54.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:54.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:54.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:54.179 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:16:54.272 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:54.318 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:54.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.325 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:16:54.340 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:54.341 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:54.341 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:54.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.341 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.0 up 2024-04-15T15:16:54.410 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-15T15:16:54.410 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:54.411 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:54.467 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-15T15:16:54.467 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:54.467 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:54.522 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:54.591 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:54.648 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:54.649 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:54.650 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:54.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.651 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:54.706 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-15T15:16:54.707 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb) 2024-04-15T15:16:54.749 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:54.777 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:54.834 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:54.834 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:54.835 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:54.836 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:54.837 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:54.837 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:54.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.837 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:54.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:2024-04-15T15:16:54.836+0000 7ff74e63e480 -1 init, newargv = 0x55aa527f0f70 newargc=15 2024-04-15T15:16:54.838 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45656]: starting ceph client 2024-04-15T15:16:54.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45656]: starting fuse 2024-04-15T15:16:54.854 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:16:54.854 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:54.854 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-15T15:16:54.855 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-15T15:16:54.856 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:54.856 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:54.856 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.1")) 2024-04-15T15:16:54.856 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-15T15:16:54.976 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.1.45656.asok 2024-04-15T15:16:54.979 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:54.986 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45656.asok 2024-04-15T15:16:54.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:54.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45656.asok status 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "1", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "45656", 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb" 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5461, 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 5461 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2974452623 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:55.126 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.1:0", 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 2974452623 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.5461 192.168.0.1:0/2974452623", 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.1:0/2974452623", 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 5, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 15, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 41, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-15T15:16:55.127 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-15T15:16:55.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.247+0000 7f6900bf4640 1 Processor -- start 2024-04-15T15:16:55.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.247+0000 7f6900bf4640 1 -- start start 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.248+0000 7f6900bf4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f68fc161fe0 0x7f68fc0d3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.248+0000 7f6900bf4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.248+0000 7f6900bf4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68fc0d5190 con 0x7f68fc0b44f0 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.248+0000 7f6900bf4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f68fc0cf3c0 con 0x7f68fc0b35f0 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.248+0000 7f6900bf4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f68fc0cf6a0 con 0x7f68fc161fe0 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46682/0 (socket says 172.21.15.70:46682) 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 -- 172.21.15.70:0/3532288577 learned_addr learned my addr 172.21.15.70:0/3532288577 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68fa59c640 1 --2- 172.21.15.70:0/3532288577 >> v2:172.21.15.96:3300/0 conn(0x7f68fc161fe0 0x7f68fc0d3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 -- 172.21.15.70:0/3532288577 >> v2:172.21.15.96:3300/0 conn(0x7f68fc161fe0 msgr2=0x7f68fc0d3320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 --2- 172.21.15.70:0/3532288577 >> v2:172.21.15.96:3300/0 conn(0x7f68fc161fe0 0x7f68fc0d3320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 -- 172.21.15.70:0/3532288577 >> v1:172.21.15.96:6789/0 conn(0x7f68fc0b44f0 legacy=0x7f68fc0d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.249+0000 7f68f9d9b640 1 -- 172.21.15.70:0/3532288577 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc0cf980 con 0x7f68fc0b35f0 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f68f9d9b640 1 --2- 172.21.15.70:0/3532288577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f68fc08cdc0 tx=0x7f68e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a13c504e7894151 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f68f959a640 1 -- 172.21.15.70:0/3532288577 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4034070 con 0x7f68fc0b35f0 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f68f959a640 1 -- 172.21.15.70:0/3532288577 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68e4030860 con 0x7f68fc0b35f0 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f68f959a640 1 -- 172.21.15.70:0/3532288577 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68e4030c30 con 0x7f68fc0b35f0 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 -- 172.21.15.70:0/3532288577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc0b35f0 msgr2=0x7f68fc0b3fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 --2- 172.21.15.70:0/3532288577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f68fc08cdc0 tx=0x7f68e402b9a0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 -- 172.21.15.70:0/3532288577 shutdown_connections 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 --2- 172.21.15.70:0/3532288577 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0b3fb0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 --2- 172.21.15.70:0/3532288577 >> v2:172.21.15.96:3300/0 conn(0x7f68fc161fe0 0x7f68fc0d3320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.250+0000 7f6900bf4640 1 -- 172.21.15.70:0/3532288577 >> 172.21.15.70:0/3532288577 conn(0x7f68fc08bf60 msgr2=0x7f68fc0d4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.251+0000 7f6900bf4640 1 -- 172.21.15.70:0/3532288577 shutdown_connections 2024-04-15T15:16:55.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.251+0000 7f6900bf4640 1 -- 172.21.15.70:0/3532288577 wait complete. 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.252+0000 7f6900bf4640 1 Processor -- start 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.252+0000 7f6900bf4640 1 -- start start 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0ce1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 0x7f68fc1f4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc0d4d60 con 0x7f68fc161fe0 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f68fc1623c0 con 0x7f68fc0b35f0 2024-04-15T15:16:55.256 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f6900bf4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f68fc0d4ee0 con 0x7f68fc0b44f0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46692/0 (socket says 172.21.15.70:46692) 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 -- 172.21.15.70:0/617399728 learned_addr learned my addr 172.21.15.70:0/617399728 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fa59c640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0ce1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68f9d9b640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 0x7f68fc1f4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 -- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 msgr2=0x7f68fc1f4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 0x7f68fc1f4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 -- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68fc0b35f0 msgr2=0x7f68fc0ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0ce1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 -- 172.21.15.70:0/617399728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68fc1f5190 con 0x7f68fc161fe0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68f9d9b640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 0x7f68fc1f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68fad9d640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f68f400c400 tx=0x7f68f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4017070 con 0x7f68fc161fe0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.253+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f40052d0 con 0x7f68fc161fe0 2024-04-15T15:16:55.257 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.254+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68fc1f5470 con 0x7f68fc161fe0 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.254+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f4003810 con 0x7f68fc161fe0 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.254+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68fc15e520 con 0x7f68fc161fe0 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.255+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68fc0689e0 con 0x7f68fc161fe0 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.255+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f68f401a070 con 0x7f68fc161fe0 2024-04-15T15:16:55.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.255+0000 7f68f2ffd640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 0x7f68dc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:55.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.256+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f68f408b110 con 0x7f68fc161fe0 2024-04-15T15:16:55.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.256+0000 7f68fa59c640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 0x7f68dc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:55.259 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.256+0000 7f68fa59c640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 0x7f68dc06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0073f0 tx=0x7f68ec00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:55.261 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.260+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f68f40593a0 con 0x7f68fc161fe0 2024-04-15T15:16:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.476+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f68fc08e100 con 0x7f68fc161fe0 2024-04-15T15:16:55.478 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.476+0000 7f68f2ffd640 1 -- 172.21.15.70:0/617399728 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f68f4058ca0 con 0x7f68fc161fe0 2024-04-15T15:16:55.478 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 msgr2=0x7f68dc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 0x7f68dc06ee40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f68ec0073f0 tx=0x7f68ec00b040 comp rx=0 tx=0).stop 2024-04-15T15:16:55.480 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 msgr2=0x7f68fc1f76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f68f400c400 tx=0x7f68f400c8c0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 shutdown_connections 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f68dc06c980 0x7f68dc06ee40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f68fc161fe0 0x7f68fc1f76c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f68fc0b44f0 0x7f68fc1f4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 --2- 172.21.15.70:0/617399728 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f68fc0b35f0 0x7f68fc0ce1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 >> 172.21.15.70:0/617399728 conn(0x7f68fc08bf60 msgr2=0x7f68fc06e420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 shutdown_connections 2024-04-15T15:16:55.481 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55.478+0000 7f6900bf4640 1 -- 172.21.15.70:0/617399728 wait complete. 2024-04-15T15:16:55.491 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:55.491 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:55.491 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi070.front.sepia.ceph.com 2024-04-15T15:16:55.491 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-15T15:16:55.492 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:55.492 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:55.492 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 2024-04-15T15:16:55.492 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:55.492 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:55.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:55.492 DEBUG:teuthology.orchestra.run.smithi070:> ip addr 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:55.509 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b2 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f0 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:d8:a4:b3 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: altname enp7s0f1 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e6 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 0c:c4:7a:8f:d3:e7 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: inet 172.21.15.70/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft 42636sec preferred_lft 42636sec 2024-04-15T15:16:55.510 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::ec4:7aff:fe8f:d3e7/64 scope link 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::c488:39ff:fe35:ded1/64 scope link 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 82:8a:03:3a:60:03 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::808a:3ff:fe3a:6003/64 scope link 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: link/ether 6a:98:ac:89:24:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:55.511 INFO:teuthology.orchestra.run.smithi070.stdout: inet6 fe80::6898:acff:fe89:2456/64 scope link 2024-04-15T15:16:55.512 INFO:teuthology.orchestra.run.smithi070.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:55.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:55.512 DEBUG:teuthology.orchestra.run.smithi070:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:55.577 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-04-15T15:16:55.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:55.579 DEBUG:teuthology.orchestra.run.smithi070:> route 2024-04-15T15:16:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:Kernel IP routing table 2024-04-15T15:16:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:55.636 INFO:teuthology.orchestra.run.smithi070.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> set -e 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:55.637 DEBUG:teuthology.orchestra.run.smithi070:> ') 2024-04-15T15:16:55.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:55.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:55.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:55.776 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:16:55.791 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:16:55.791 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:16:55.792 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.2' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:55.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:55.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link set brx.1 up 2024-04-15T15:16:55.861 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-15T15:16:55.861 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:55.861 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:55.916 INFO:teuthology.orchestra.run.smithi070.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-15T15:16:55.916 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:55.916 DEBUG:teuthology.orchestra.run.smithi070:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-15T15:16:55.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:56.041 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:56.097 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:56.097 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:56.097 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:56.098 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:56.099 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:56.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:56.100 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:56.157 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:16:56.157 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-15T15:16:56.158 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb) 2024-04-15T15:16:56.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo modprobe fuse 2024-04-15T15:16:56.228 DEBUG:teuthology.orchestra.run.smithi070:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:56.284 INFO:teuthology.orchestra.run.smithi070.stdout:/proc 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/dev 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/security 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/shm 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/pts 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/run 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/cgroup 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/pstore 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/bpf 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/ 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/selinux 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:56.285 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/hugepages 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/debug 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/dev/mqueue 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/tracing 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/sys/kernel/config 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/run/user/1000 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-0 2024-04-15T15:16:56.286 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-1 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-2 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/var/lib/ceph/osd/ceph-3 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run.smithi070.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-15T15:16:56.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:56.287 DEBUG:teuthology.orchestra.run.smithi070:> ls /sys/fs/fuse/connections 2024-04-15T15:16:56.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:2024-04-15T15:16:56.285+0000 7f03d003a480 -1 init, newargv = 0x55eacb02eb30 newargc=15 2024-04-15T15:16:56.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[55717]: starting ceph client 2024-04-15T15:16:56.294 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[55717]: starting fuse 2024-04-15T15:16:56.304 INFO:teuthology.orchestra.run.smithi070.stdout:43 2024-04-15T15:16:56.304 INFO:teuthology.orchestra.run.smithi070.stdout:44 2024-04-15T15:16:56.304 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-15T15:16:56.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> import glob 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> import re 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> import os 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> import subprocess 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> def _find_admin_socket(client_name): 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> files = glob.glob(asok_path) 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> # Given a non-glob path, it better be there 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> if "*" not in asok_path: 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> assert(len(files) == 1) 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> return files[0] 2024-04-15T15:16:56.305 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> for f in files: 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> contents = proc_f.read() 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> if mountpoint in contents: 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> return f 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> print(_find_admin_socket("client.2")) 2024-04-15T15:16:56.306 DEBUG:teuthology.orchestra.run.smithi070:> ' 2024-04-15T15:16:56.428 INFO:teuthology.orchestra.run.smithi070.stdout:/var/run/ceph/ceph-client.2.55717.asok 2024-04-15T15:16:56.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:56.439 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.55717.asok 2024-04-15T15:16:56.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:56.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.55717.asok status 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "metadata": { 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "entity_id": "2", 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "hostname": "smithi070", 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-15T15:16:56.578 INFO:teuthology.orchestra.run.smithi070.stdout: "pid": "55717", 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "root": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb" 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_count": 0, 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5964, 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "inst": { 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "name": { 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "client", 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 5964 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:56.579 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2000636358 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": { 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "v1", 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "nonce": 2000636358 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "inst_str": "client.5964 192.168.0.2:0/2000636358", 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "addr_str": "192.168.0.2:0/2000636358", 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "inode_count": 5, 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "mds_epoch": 15, 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch": 41, 2024-04-15T15:16:56.580 INFO:teuthology.orchestra.run.smithi070.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:56.581 INFO:teuthology.orchestra.run.smithi070.stdout: "blocklisted": false, 2024-04-15T15:16:56.581 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs" 2024-04-15T15:16:56.581 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:56.586 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-15T15:16:56.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.688+0000 7f54dcb8d640 1 Processor -- start 2024-04-15T15:16:56.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.688+0000 7f54dcb8d640 1 -- start start 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54dcb8d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54d8158de0 0x7f54d806f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54dcb8d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d80aee30 con 0x7f54d80ae7d0 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54dcb8d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54d80af110 con 0x7f54d806fbc0 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54dcb8d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f54d8070030 con 0x7f54d8158de0 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f54d8158de0 0x7f54d806f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d6d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f54d80ae7d0 0x7f54d8071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34710/0 (socket says 172.21.15.70:34710) 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d6d9d640 1 -- 172.21.15.70:0/2841166270 learned_addr learned my addr 172.21.15.70:0/2841166270 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d659c640 1 -- 172.21.15.70:0/2841166270 >> v1:172.21.15.96:6789/0 conn(0x7f54d806fbc0 legacy=0x7f54d80ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:56.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d5d9b640 1 -- 172.21.15.70:0/2841166270 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578380313 0 0) 0x7f54d80aee30 con 0x7f54d80ae7d0 2024-04-15T15:16:56.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d659c640 1 -- 172.21.15.70:0/2841166270 >> v1:172.21.15.70:6789/0 conn(0x7f54d80ae7d0 legacy=0x7f54d8071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.689+0000 7f54d659c640 1 -- 172.21.15.70:0/2841166270 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d80af3f0 con 0x7f54d8158de0 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54d659c640 1 --2- 172.21.15.70:0/2841166270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d8158de0 0x7f54d806f680 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f54c80097a0 tx=0x7f54c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89a16c67e324e6c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54d5d9b640 1 -- 172.21.15.70:0/2841166270 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c80315a0 con 0x7f54d8158de0 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54d5d9b640 1 -- 172.21.15.70:0/2841166270 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c8037070 con 0x7f54d8158de0 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54d5d9b640 1 -- 172.21.15.70:0/2841166270 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c8031960 con 0x7f54d8158de0 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54dcb8d640 1 -- 172.21.15.70:0/2841166270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d8158de0 msgr2=0x7f54d806f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.690+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/2841166270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d8158de0 0x7f54d806f680 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f54c80097a0 tx=0x7f54c802b840 comp rx=0 tx=0).stop 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.691+0000 7f54dcb8d640 1 -- 172.21.15.70:0/2841166270 shutdown_connections 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.691+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/2841166270 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d8158de0 0x7f54d806f680 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.691+0000 7f54dcb8d640 1 -- 172.21.15.70:0/2841166270 >> 172.21.15.70:0/2841166270 conn(0x7f54d80b1f60 msgr2=0x7f54d80b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.691+0000 7f54dcb8d640 1 -- 172.21.15.70:0/2841166270 shutdown_connections 2024-04-15T15:16:56.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.691+0000 7f54dcb8d640 1 -- 172.21.15.70:0/2841166270 wait complete. 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.692+0000 7f54dcb8d640 1 Processor -- start 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.692+0000 7f54dcb8d640 1 -- start start 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 0x7f54d806f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 0x7f54d815f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f54d81591c0 con 0x7f54d80ae7d0 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f54d80af9b0 con 0x7f54d8158de0 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54dcb8d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f54d8159340 con 0x7f54d806fbc0 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46714/0 (socket says 172.21.15.70:46714) 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 -- 172.21.15.70:0/589562134 learned_addr learned my addr 172.21.15.70:0/589562134 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:56.696 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54d6d9d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 0x7f54d815f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54d659c640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 0x7f54d806f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 -- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 msgr2=0x7f54d806f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 0x7f54d806f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 -- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 msgr2=0x7f54d815f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 0x7f54d815f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 -- 172.21.15.70:0/589562134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54d80b1b90 con 0x7f54d80ae7d0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54d6d9d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 0x7f54d815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54d659c640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 0x7f54d806f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cdd9b640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f54c000c430 tx=0x7f54c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.693+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0011790 con 0x7f54d80ae7d0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.694+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54c0017070 con 0x7f54d80ae7d0 2024-04-15T15:16:56.697 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.694+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54d809ac80 con 0x7f54d80ae7d0 2024-04-15T15:16:56.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.694+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0011b70 con 0x7f54d80ae7d0 2024-04-15T15:16:56.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.694+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54d806e9d0 con 0x7f54d80ae7d0 2024-04-15T15:16:56.698 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.694+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54d80689e0 con 0x7f54d80ae7d0 2024-04-15T15:16:56.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.695+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f54c0011d10 con 0x7f54d80ae7d0 2024-04-15T15:16:56.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.695+0000 7f54cf7fe640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 0x7f54b806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:56.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.696+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f54c008bb00 con 0x7f54d80ae7d0 2024-04-15T15:16:56.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.696+0000 7f54d659c640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 0x7f54b806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:56.699 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.696+0000 7f54d659c640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 0x7f54b806ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f54c8036000 tx=0x7f54c8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:56.702 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.700+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f54c0059e20 con 0x7f54d80ae7d0 2024-04-15T15:16:56.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.913+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f54d806e060 con 0x7f54d80ae7d0 2024-04-15T15:16:56.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.914+0000 7f54cf7fe640 1 -- 172.21.15.70:0/589562134 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f54c0059720 con 0x7f54d80ae7d0 2024-04-15T15:16:56.916 INFO:teuthology.orchestra.run.smithi070.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 msgr2=0x7f54b806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 0x7f54b806ee40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f54c8036000 tx=0x7f54c8008030 comp rx=0 tx=0).stop 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 msgr2=0x7f54d809a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f54c000c430 tx=0x7f54c000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 shutdown_connections 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f54d8158de0 0x7f54d815f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f54b806c980 0x7f54b806ee40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f54d80ae7d0 0x7f54d809a250 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.918 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 --2- 172.21.15.70:0/589562134 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f54d806fbc0 0x7f54d806f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:56.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 >> 172.21.15.70:0/589562134 conn(0x7f54d80b1f60 msgr2=0x7f54d8098190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:56.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 shutdown_connections 2024-04-15T15:16:56.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:56.916+0000 7f54dcb8d640 1 -- 172.21.15.70:0/589562134 wait complete. 2024-04-15T15:16:56.928 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-15T15:16:56.928 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-15T15:16:56.928 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi096.front.sepia.ceph.com 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-15T15:16:56.929 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:56.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:56.929 DEBUG:teuthology.orchestra.run.smithi096:> ip addr 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: inet 127.0.0.1/8 scope host lo 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 ::1/128 scope host 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:56.949 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bc brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f0 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:d8:d9:bd brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: altname enp7s0f1 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0c brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether 0c:c4:7a:8f:cc:0d brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: inet 172.21.15.96/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft 42608sec preferred_lft 42608sec 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::ec4:7aff:fe8f:cc0d/64 scope link 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-15T15:16:56.950 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::c27:e4ff:fe37:c79c/64 scope link 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether f6:ed:c4:10:c9:d9 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::f4ed:c4ff:fe10:c9d9/64 scope link 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: link/ether d2:96:56:49:53:76 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: inet6 fe80::d096:56ff:fe49:5376/64 scope link 2024-04-15T15:16:56.951 INFO:teuthology.orchestra.run.smithi096.stdout: valid_lft forever preferred_lft forever 2024-04-15T15:16:56.952 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:56.952 DEBUG:teuthology.orchestra.run.smithi096:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-15T15:16:57.022 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-04-15T15:16:57.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.024 DEBUG:teuthology.orchestra.run.smithi096:> route 2024-04-15T15:16:57.083 INFO:teuthology.orchestra.run.smithi096.stdout:Kernel IP routing table 2024-04-15T15:16:57.083 INFO:teuthology.orchestra.run.smithi096.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-15T15:16:57.083 INFO:teuthology.orchestra.run.smithi096.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-15T15:16:57.083 INFO:teuthology.orchestra.run.smithi096.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-15T15:16:57.084 INFO:teuthology.orchestra.run.smithi096.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> set -e 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-15T15:16:57.084 DEBUG:teuthology.orchestra.run.smithi096:> ') 2024-04-15T15:16:57.179 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-15T15:16:57.225 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:57.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.231 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:16:57.247 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:16:57.247 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:16:57.247 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.3' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:57.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.248 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link set brx.1 up 2024-04-15T15:16:57.317 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-15T15:16:57.317 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:57.318 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:57.374 INFO:teuthology.orchestra.run.smithi096.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-15T15:16:57.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-15T15:16:57.374 DEBUG:teuthology.orchestra.run.smithi096:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-15T15:16:57.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:57.499 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:57.555 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:57.555 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:57.555 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:57.556 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:57.557 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:57.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.558 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:57.611 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:16:57.611 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-04-15T15:16:57.611 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3 --client_mountpoint=/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb) 2024-04-15T15:16:57.655 DEBUG:teuthology.orchestra.run.smithi096:> sudo modprobe fuse 2024-04-15T15:16:57.683 DEBUG:teuthology.orchestra.run.smithi096:> cat /proc/self/mounts | awk '{print $2}' 2024-04-15T15:16:57.739 INFO:teuthology.orchestra.run.smithi096.stdout:/proc 2024-04-15T15:16:57.739 INFO:teuthology.orchestra.run.smithi096.stdout:/sys 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/dev 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/security 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/shm 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/pts 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/run 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/cgroup 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/pstore 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/bpf 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/ 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/selinux 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/proc/sys/fs/binfmt_misc 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/mqueue 2024-04-15T15:16:57.740 INFO:teuthology.orchestra.run.smithi096.stdout:/dev/hugepages 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/debug 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/tracing 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/fs/fuse/connections 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/sys/kernel/config 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-sysctl.service 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/nfs/rpc_pipefs 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/run/user/1000 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-4 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-5 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-6 2024-04-15T15:16:57.741 INFO:teuthology.orchestra.run.smithi096.stdout:/var/lib/ceph/osd/ceph-7 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run.smithi096.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-15T15:16:57.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.742 DEBUG:teuthology.orchestra.run.smithi096:> ls /sys/fs/fuse/connections 2024-04-15T15:16:57.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:2024-04-15T15:16:57.742+0000 7f85183fd480 -1 init, newargv = 0x55a7d0b05ff0 newargc=15 2024-04-15T15:16:57.743 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:ceph-fuse[45996]: starting ceph client 2024-04-15T15:16:57.750 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:ceph-fuse[45996]: starting fuse 2024-04-15T15:16:57.758 INFO:teuthology.orchestra.run.smithi096.stdout:42 2024-04-15T15:16:57.759 INFO:teuthology.orchestra.run.smithi096.stdout:44 2024-04-15T15:16:57.759 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> sudo stdin-killer -- python3 -c ' 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> import glob 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> import re 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> import os 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> import subprocess 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:57.759 DEBUG:teuthology.orchestra.run.smithi096:> def _find_admin_socket(client_name): 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> files = glob.glob(asok_path) 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> # Given a non-glob path, it better be there 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> if "*" not in asok_path: 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> assert(len(files) == 1) 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> return files[0] 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> for f in files: 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> contents = proc_f.read() 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> if mountpoint in contents: 2024-04-15T15:16:57.760 DEBUG:teuthology.orchestra.run.smithi096:> return f 2024-04-15T15:16:57.761 DEBUG:teuthology.orchestra.run.smithi096:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-15T15:16:57.761 DEBUG:teuthology.orchestra.run.smithi096:> 2024-04-15T15:16:57.761 DEBUG:teuthology.orchestra.run.smithi096:> print(_find_admin_socket("client.3")) 2024-04-15T15:16:57.761 DEBUG:teuthology.orchestra.run.smithi096:> ' 2024-04-15T15:16:57.880 INFO:teuthology.orchestra.run.smithi096.stdout:/var/run/ceph/ceph-client.3.45996.asok 2024-04-15T15:16:57.884 INFO:teuthology.orchestra.run.smithi096.stderr:2024-04-15T15:16:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:57.891 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.45996.asok 2024-04-15T15:16:57.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:57.892 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.45996.asok status 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "metadata": { 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_sha1": "9bf1c4ba967dd24077be6e5b87edda4ca87a6e14", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "ceph_version": "ceph version 19.0.0-2861-g9bf1c4ba (9bf1c4ba967dd24077be6e5b87edda4ca87a6e14) squid (dev)", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "entity_id": "3", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "hostname": "smithi096", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "pid": "45996", 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "root": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb" 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:58.030 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_count": 0, 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "dentry_pinned_count": 0, 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "id": 5126, 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "inst": { 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "name": { 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "client", 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "num": 5126 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 4170219554 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-04-15T15:16:58.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": { 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "type": "v1", 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "addr": "192.168.0.2:0", 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "nonce": 4170219554 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "inst_str": "client.5126 192.168.0.2:0/4170219554", 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "addr_str": "192.168.0.2:0/4170219554", 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "inode_count": 5, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "mds_epoch": 15, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch": 41, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "osd_epoch_barrier": 0, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "blocklisted": false, 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout: "fs_name": "cephfs" 2024-04-15T15:16:58.032 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-04-15T15:16:58.038 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'touch file') 2024-04-15T15:16:58.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-15T15:16:58.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-15T15:16:58.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:16:58.110 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quiesce.block -v 1 file) 2024-04-15T15:16:58.133 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 quiesce path /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb --wait 2024-04-15T15:16:58.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.278+0000 7f556d1d5640 1 Processor -- start 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.278+0000 7f556d1d5640 1 -- start start 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556d1d5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55680cc6b0 0x7f55680b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556d1d5640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 0x7f55680d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556d1d5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55680b38d0 con 0x7f55680ce770 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556d1d5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f55680b36a0 con 0x7f55680cc6b0 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556d1d5640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f55680b3bb0 con 0x7f55680d0470 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f55680cc6b0 0x7f55680b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5566d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f55680ce770 0x7f556815c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34716/0 (socket says 172.21.15.70:34716) 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5566d9d640 1 -- 172.21.15.70:0/3861689060 learned_addr learned my addr 172.21.15.70:0/3861689060 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556759e640 1 --2- 172.21.15.70:0/3861689060 >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 0x7f55680d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 -- 172.21.15.70:0/3861689060 >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 msgr2=0x7f55680d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.283 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 --2- 172.21.15.70:0/3861689060 >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 0x7f55680d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 -- 172.21.15.70:0/3861689060 >> v1:172.21.15.70:6789/0 conn(0x7f55680ce770 legacy=0x7f556815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 -- 172.21.15.70:0/3861689060 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55680b3e90 con 0x7f55680cc6b0 2024-04-15T15:16:58.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556759e640 1 --2- 172.21.15.70:0/3861689060 >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 0x7f55680d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:58.284 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5565d9b640 1 -- 172.21.15.70:0/3861689060 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1352353643 0 0) 0x7f55680b38d0 con 0x7f55680ce770 2024-04-15T15:16:58.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f556659c640 1 --2- 172.21.15.70:0/3861689060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680cc6b0 0x7f55680b3160 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f55680cfce0 tx=0x7f555c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec35e0e9d9ae8612 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5565d9b640 1 -- 172.21.15.70:0/3861689060 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0315a0 con 0x7f55680cc6b0 2024-04-15T15:16:58.285 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5565d9b640 1 -- 172.21.15.70:0/3861689060 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c037070 con 0x7f55680cc6b0 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.280+0000 7f5565d9b640 1 -- 172.21.15.70:0/3861689060 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c031960 con 0x7f55680cc6b0 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 -- 172.21.15.70:0/3861689060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680cc6b0 msgr2=0x7f55680b3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 --2- 172.21.15.70:0/3861689060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680cc6b0 0x7f55680b3160 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f55680cfce0 tx=0x7f555c02b840 comp rx=0 tx=0).stop 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 -- 172.21.15.70:0/3861689060 shutdown_connections 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 --2- 172.21.15.70:0/3861689060 >> v2:172.21.15.96:3300/0 conn(0x7f55680d0470 0x7f55680d0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 --2- 172.21.15.70:0/3861689060 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680cc6b0 0x7f55680b3160 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 -- 172.21.15.70:0/3861689060 >> 172.21.15.70:0/3861689060 conn(0x7f556806bbc0 msgr2=0x7f556806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 -- 172.21.15.70:0/3861689060 shutdown_connections 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.281+0000 7f556d1d5640 1 -- 172.21.15.70:0/3861689060 wait complete. 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.282+0000 7f556d1d5640 1 Processor -- start 2024-04-15T15:16:58.286 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.282+0000 7f556d1d5640 1 -- start start 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55680cc6b0 0x7f5568146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 0x7f55680ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f556815c7d0 con 0x7f55680d0470 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55680ceb50 con 0x7f55680ce770 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556d1d5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f556815c950 con 0x7f55680cc6b0 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46736/0 (socket says 172.21.15.70:46736) 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.283+0000 7f556759e640 1 -- 172.21.15.70:0/2103151669 learned_addr learned my addr 172.21.15.70:0/2103151669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556659c640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 0x7f55680ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f5566d9d640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55680cc6b0 0x7f5568146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 -- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55680cc6b0 msgr2=0x7f5568146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55680cc6b0 0x7f5568146690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 -- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 msgr2=0x7f55680ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 0x7f55680ae7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 -- 172.21.15.70:0/2103151669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55681523d0 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556659c640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 0x7f55680ae7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556759e640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f556000c590 tx=0x7f556000ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f554f7fe640 1 -- 172.21.15.70:0/2103151669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560015020 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f554f7fe640 1 -- 172.21.15.70:0/2103151669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5560004440 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55681520f0 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f554f7fe640 1 -- 172.21.15.70:0/2103151669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560003850 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.284+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55680adfb0 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.285+0000 7f554f7fe640 1 -- 172.21.15.70:0/2103151669 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f5560018020 con 0x7f55680d0470 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.286+0000 7f554f7fe640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 0x7f555406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.286+0000 7f554f7fe640 1 -- 172.21.15.70:0/2103151669 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f556008bd90 con 0x7f55680d0470 2024-04-15T15:16:58.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.286+0000 7f5566d9d640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 0x7f555406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.286+0000 7f5566d9d640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 0x7f555406ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f55580012e0 tx=0x7f5558009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.287+0000 7f556fdb6740 1 Processor -- start 2024-04-15T15:16:58.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.287+0000 7f556fdb6740 1 -- start start 2024-04-15T15:16:58.299 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdcddde20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdcdde360 0x562bdce0e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562bdcda5ad0 con 0x562bdce0b260 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562bdcd49560 con 0x562bdcdde360 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562bdccefc00 con 0x562bdce0ae80 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46740/0 (socket says 172.21.15.70:46740) 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 -- 172.21.15.70:0/2218861333 learned_addr learned my addr 172.21.15.70:0/2218861333 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f5566d9d640 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdcddde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556759e640 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdcdde360 0x562bdce0e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 -- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 msgr2=0x562bdcddde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdcddde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 -- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdcdde360 msgr2=0x562bdce0e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.300 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdcdde360 0x562bdce0e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 -- 172.21.15.70:0/2218861333 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bdcb28030 con 0x562bdce0b260 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f556659c640 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f555c000c00 tx=0x7f555c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96f91d8f9e256062 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f554cff9640 1 -- 172.21.15.70:0/2218861333 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c0319a0 con 0x562bdce0b260 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f554cff9640 1 -- 172.21.15.70:0/2218861333 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c037040 con 0x562bdce0b260 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.288+0000 7f554cff9640 1 -- 172.21.15.70:0/2218861333 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c031d40 con 0x562bdce0b260 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 -- 172.21.15.70:0/2218861333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 msgr2=0x562bdce0dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f555c000c00 tx=0x7f555c030000 comp rx=0 tx=0).stop 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 -- 172.21.15.70:0/2218861333 shutdown_connections 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdcdde360 0x562bdce0e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0b260 0x562bdce0dc10 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 --2- 172.21.15.70:0/2218861333 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdcddde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 -- 172.21.15.70:0/2218861333 >> 172.21.15.70:0/2218861333 conn(0x562bdcde0090 msgr2=0x562bdce28c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 -- 172.21.15.70:0/2218861333 shutdown_connections 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.289+0000 7f556fdb6740 1 -- 172.21.15.70:0/2218861333 wait complete. 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.290+0000 7f556fdb6740 1 -- start start 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.290+0000 7f556fdb6740 1 Processor -- start 2024-04-15T15:16:58.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdce29020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0b260 0x562bdce13f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x562bdcd03110 con 0x562bdce08500 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562bdcd45520 con 0x562bdce0b260 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562bdcde2730 con 0x562bdce0ae80 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 10 client.0 _ll_register_callbacks cb 0x562bdca5b410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46754/0 (socket says 172.21.15.70:46754) 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 -- 172.21.15.70:0/200759840 learned_addr learned my addr 172.21.15.70:0/200759840 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f5566d9d640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdce29020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556659c640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0b260 0x562bdce13f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 -- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 msgr2=0x562bdce29020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdce29020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 -- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0b260 msgr2=0x562bdce13f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0b260 0x562bdce13f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 -- 172.21.15.70:0/200759840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bdca5bc20 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556759e640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f556000cf50 tx=0x7f556005dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2fc4a5b36fdca45 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f5545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5560087030 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f5560062200 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f555c037040 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f5560021b80 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 10 client.5988 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.291+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x562bdca1b920 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f555c037040 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f5560058cf0 con 0x562bdce08500 2024-04-15T15:16:58.303 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 10 client.5988 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 10 client.5988 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 20 client.5988 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce0fbb0 0x562bdce0ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 4 client.5988 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bdcd56f80 con 0x562bdce0fbb0 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f5566d9d640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce0fbb0 0x562bdce0ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f5545ffb640 10 client.5988 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.292+0000 7f5566d9d640 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce0fbb0 0x562bdce0ffc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=485e4b732bdef912 server_cookie=fec729190f9dc559 in_seq=0 out_seq=0 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f5545ffb640 1 -- 172.21.15.70:0/200759840 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bdcd56f80 con 0x562bdce0fbb0 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f5545ffb640 10 client.5988 handle_command_reply: tid=0 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 client.5988 shutdown 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 msgr2=0x562bdce144a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f556000cf50 tx=0x7f556005dcd0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 shutdown_connections 2024-04-15T15:16:58.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce144a0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0b260 0x562bdce13f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce0fbb0 0x562bdce0ffc0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 --2- 172.21.15.70:0/200759840 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0ae80 0x562bdce29020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f5545ffb640 0 client.5988 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 >> 172.21.15.70:0/200759840 conn(0x562bdcdde360 msgr2=0x562bdcde4c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 shutdown_connections 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 1 -- 172.21.15.70:0/200759840 wait complete. 2024-04-15T15:16:58.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.293+0000 7f556fdb6740 20 client.5988 trim_cache size 0 max 16384 2024-04-15T15:16:58.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.310+0000 7f556fdb6740 1 Processor -- start 2024-04-15T15:16:58.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.310+0000 7f556fdb6740 1 -- start start 2024-04-15T15:16:58.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 0x562bdce9c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce2c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5568200fb0 con 0x562bdce08500 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f55681fffe0 con 0x562bdce0ae80 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f55681f4ae0 con 0x562bdce0b260 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46758/0 (socket says 172.21.15.70:46758) 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 -- 172.21.15.70:0/3036831838 learned_addr learned my addr 172.21.15.70:0/3036831838 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556759e640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce2c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556659c640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 0x562bdce9c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 -- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 msgr2=0x562bdce2c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce2c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 -- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 msgr2=0x562bdce9c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 0x562bdce9c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 -- 172.21.15.70:0/3036831838 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bdce11bd0 con 0x562bdce08500 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f556659c640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 0x562bdce9c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:58.316 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5566d9d640 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f55580019f0 tx=0x7f5558010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768316035629e952 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5547fff640 1 -- 172.21.15.70:0/3036831838 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55580271d0 con 0x562bdce08500 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5547fff640 1 -- 172.21.15.70:0/3036831838 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5558027370 con 0x562bdce08500 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.311+0000 7f5547fff640 1 -- 172.21.15.70:0/3036831838 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5558027720 con 0x562bdce08500 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 -- 172.21.15.70:0/3036831838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 msgr2=0x562bdce07800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f55580019f0 tx=0x7f5558010000 comp rx=0 tx=0).stop 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 -- 172.21.15.70:0/3036831838 shutdown_connections 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce2c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce0ae80 0x562bdce9c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 --2- 172.21.15.70:0/3036831838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce08500 0x562bdce07800 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 -- 172.21.15.70:0/3036831838 >> 172.21.15.70:0/3036831838 conn(0x562bdcdde360 msgr2=0x562bdce2af70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 -- 172.21.15.70:0/3036831838 shutdown_connections 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.312+0000 7f556fdb6740 1 -- 172.21.15.70:0/3036831838 wait complete. 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 -- start start 2024-04-15T15:16:58.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 Processor -- start 2024-04-15T15:16:58.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce08500 0x562bdce2a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.320 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce09160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f55681ffa60 con 0x562bdce0ae80 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x562bdcd4ba70 con 0x562bdce08500 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x562bdce119b0 con 0x562bdce0b260 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556fdb6740 10 client.0 _ll_register_callbacks cb 0x7f5560059970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556659c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46774/0 (socket says 172.21.15.70:46774) 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.313+0000 7f556659c640 1 -- 172.21.15.70:0/246719459 learned_addr learned my addr 172.21.15.70:0/246719459 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5566d9d640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce08500 0x562bdce2a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556759e640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce09160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 -- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 msgr2=0x562bdce09160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce09160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 -- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce08500 msgr2=0x562bdce2a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce08500 0x562bdce2a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 -- 172.21.15.70:0/246719459 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bdce2ca60 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556659c640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f555c02be50 tx=0x7f555c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab7ef04587ac1dc server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f555c055070 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f555c031930 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5558027370 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f555c02bec0 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 10 client.5994 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x562bdce0a1a0 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5558027370 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f555c058020 con 0x562bdce0ae80 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 10 client.5994 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:58.322 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 10 client.5994 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 20 client.5994 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce12540 0x562bdce12950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 4 client.5994 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb", "wait": true}] 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.314+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "quiesce path", "path": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb", "wait": true}) -- 0x562bdce15cc0 con 0x562bdce12540 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.315+0000 7f5566d9d640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce12540 0x562bdce12950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.315+0000 7f5545ffb640 10 client.5994 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:58.323 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.315+0000 7f5566d9d640 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce12540 0x562bdce12950 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd41562f24010e25 server_cookie=cc1ea6e1b045038c in_seq=0 out_seq=0 2024-04-15T15:16:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f5545ffb640 1 -- 172.21.15.70:0/246719459 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+2332 (crc 0 0 0) 0x562bdce15cc0 con 0x562bdce12540 2024-04-15T15:16:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f5545ffb640 10 client.5994 handle_command_reply: tid=0 2024-04-15T15:16:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 client.5994 shutdown 2024-04-15T15:16:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 msgr2=0x562bdce2c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f555c02be50 tx=0x7f555c004730 comp rx=0 tx=0).stop 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 shutdown_connections 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x562bdce12540 0x562bdce12950 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x562bdce0b260 0x562bdce09160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f5545ffb640 0 client.5994 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x562bdce0ae80 0x562bdce2c520 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 --2- 172.21.15.70:0/246719459 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x562bdce08500 0x562bdce2a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 >> 172.21.15.70:0/246719459 conn(0x562bdcdde360 msgr2=0x562bdce13e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 shutdown_connections 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 1 -- 172.21.15.70:0/246719459 wait complete. 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.319+0000 7f556fdb6740 20 client.5994 trim_cache size 0 max 16384 2024-04-15T15:16:58.325 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "op": { 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "result": 0, 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "quiesce complete", 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "reqid": { 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "entity": { 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type": "mds", 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "num": 0 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 1 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "op_type": "internal_op", 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "internal_op": 5384, 2024-04-15T15:16:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "op_name": "quiesce_path", 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317638+0000", 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated" 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317638+0000", 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read" 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317638+0000", 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled" 2024-04-15T15:16:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317638+0000", 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read" 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317638+0000", 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched" 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317694+0000", 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317713+0000", 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317730+0000", 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.317750+0000", 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.320915+0000", 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.320926+0000", 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.320936+0000", 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.320946+0000", 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "acquired locks" 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-04-15T15:16:58.320959+0000", 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "quiesce complete" 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "locks": null 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "state": { 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes": 2, 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_quiesced": 1, 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_blocked": 1, 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "inodes_dropped": 0, 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "failed": [] 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-04-15T15:16:58.331 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:58.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 msgr2=0x7f555406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 0x7f555406ee40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f55580012e0 tx=0x7f5558009040 comp rx=0 tx=0).stop 2024-04-15T15:16:58.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 msgr2=0x7f5568151630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f556000c590 tx=0x7f556000ca50 comp rx=0 tx=0).stop 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 shutdown_connections 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f555406c980 0x7f555406ee40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f55680d0470 0x7f5568151630 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f55680ce770 0x7f55680ae7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 --2- 172.21.15.70:0/2103151669 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f55680cc6b0 0x7f5568146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 >> 172.21.15.70:0/2103151669 conn(0x7f556806bbc0 msgr2=0x7f556806dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 shutdown_connections 2024-04-15T15:16:58.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.320+0000 7f556d1d5640 1 -- 172.21.15.70:0/2103151669 wait complete. 2024-04-15T15:16:58.334 DEBUG:tasks.cephfs.test_quiesce:{'op': {'result': 0, 'flag_point': 'quiesce complete', 'reqid': {'entity': {'type': 'mds', 'num': 0}, 'tid': 1}, 'op_type': 'internal_op', 'internal_op': 5384, 'op_name': 'quiesce_path', 'events': [{'time': '2024-04-15T15:16:58.317638+0000', 'event': 'initiated'}, {'time': '2024-04-15T15:16:58.317638+0000', 'event': 'header_read'}, {'time': '2024-04-15T15:16:58.317638+0000', 'event': 'throttled'}, {'time': '2024-04-15T15:16:58.317638+0000', 'event': 'all_read'}, {'time': '2024-04-15T15:16:58.317638+0000', 'event': 'dispatched'}, {'time': '2024-04-15T15:16:58.317694+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.317713+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.317730+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.317750+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.320915+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.320926+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.320936+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.320946+0000', 'event': 'acquired locks'}, {'time': '2024-04-15T15:16:58.320959+0000', 'event': 'quiesce complete'}], 'locks': None}, 'state': {'inodes': 2, 'inodes_quiesced': 1, 'inodes_blocked': 1, 'inodes_dropped': 0, 'failed': []}} 2024-04-15T15:16:58.334 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:16:58.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.435+0000 7fa2a8750640 1 Processor -- start 2024-04-15T15:16:58.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.435+0000 7fa2a8750640 1 -- start start 2024-04-15T15:16:58.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a8750640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2a015e700 0x7fa2a00b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a8750640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 0x7fa2a00d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a8750640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2a00b47a0 con 0x7fa2a015eae0 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a8750640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa2a00b53b0 con 0x7fa2a015e700 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a8750640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa2a00b4a80 con 0x7fa2a00d2360 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a64ec640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2a015e700 0x7fa2a00b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a6ced640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa2a015eae0 0x7fa2a00cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34732/0 (socket says 172.21.15.70:34732) 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.436+0000 7fa2a6ced640 1 -- 172.21.15.70:0/1136549792 learned_addr learned my addr 172.21.15.70:0/1136549792 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a5ceb640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 0x7fa2a00d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a64ec640 1 -- 172.21.15.70:0/1136549792 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 msgr2=0x7fa2a00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a64ec640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 0x7fa2a00d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a64ec640 1 -- 172.21.15.70:0/1136549792 >> v1:172.21.15.70:6789/0 conn(0x7fa2a015eae0 legacy=0x7fa2a00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a64ec640 1 -- 172.21.15.70:0/1136549792 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a00b4d60 con 0x7fa2a015e700 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a54ea640 1 -- 172.21.15.70:0/1136549792 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 5887575 0 0) 0x7fa2a00b47a0 con 0x7fa2a015eae0 2024-04-15T15:16:58.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a5ceb640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 0x7fa2a00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.437+0000 7fa2a64ec640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a015e700 0x7fa2a00b5840 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009de0 tx=0x7fa29c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf17dd1a7313305 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a54ea640 1 -- 172.21.15.70:0/1136549792 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c039070 con 0x7fa2a015e700 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a54ea640 1 -- 172.21.15.70:0/1136549792 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c02cc90 con 0x7fa2a015e700 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a54ea640 1 -- 172.21.15.70:0/1136549792 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c034660 con 0x7fa2a015e700 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 -- 172.21.15.70:0/1136549792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a015e700 msgr2=0x7fa2a00b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a015e700 0x7fa2a00b5840 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009de0 tx=0x7fa29c02bee0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 -- 172.21.15.70:0/1136549792 shutdown_connections 2024-04-15T15:16:58.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a00d2360 0x7fa2a00d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 --2- 172.21.15.70:0/1136549792 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a015e700 0x7fa2a00b5840 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 -- 172.21.15.70:0/1136549792 >> 172.21.15.70:0/1136549792 conn(0x7fa2a008bf60 msgr2=0x7fa2a00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 -- 172.21.15.70:0/1136549792 shutdown_connections 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.438+0000 7fa2a8750640 1 -- 172.21.15.70:0/1136549792 wait complete. 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.439+0000 7fa2a8750640 1 Processor -- start 2024-04-15T15:16:58.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.439+0000 7fa2a8750640 1 -- start start 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a015e700 0x7fa2a00b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2a015eae0 0x7fa2a00b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a00b5d80 con 0x7fa2a00d2360 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2a00b4450 con 0x7fa2a015eae0 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a8750640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa2a00d1d40 con 0x7fa2a015e700 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46786/0 (socket says 172.21.15.70:46786) 2024-04-15T15:16:58.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 -- 172.21.15.70:0/946563902 learned_addr learned my addr 172.21.15.70:0/946563902 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a6ced640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2a015eae0 0x7fa2a00b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 -- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a015e700 msgr2=0x7fa2a00b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a015e700 0x7fa2a00b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 -- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2a015eae0 msgr2=0x7fa2a00b90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2a015eae0 0x7fa2a00b90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.440+0000 7fa2a64ec640 1 -- 172.21.15.70:0/946563902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2a014b5d0 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa2a64ec640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009de0 tx=0x7fa29c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c039050 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa29c045070 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2a00c8010 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29c03e850 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.441+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2a014b8b0 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.442+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2a00689e0 con 0x7fa2a00d2360 2024-04-15T15:16:58.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.442+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa29c03e9f0 con 0x7fa2a00d2360 2024-04-15T15:16:58.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.443+0000 7fa296ffd640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 0x7fa28406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.443+0000 7fa2a5ceb640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 0x7fa28406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.443+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7fa29c0aea10 con 0x7fa2a00d2360 2024-04-15T15:16:58.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.443+0000 7fa2a5ceb640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 0x7fa28406ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa290006f00 tx=0x7fa290008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.450 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.447+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa29c032e20 con 0x7fa2a00d2360 2024-04-15T15:16:58.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.660+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa2a00b1280 con 0x7fa2a00d2360 2024-04-15T15:16:58.663 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.660+0000 7fa296ffd640 1 -- 172.21.15.70:0/946563902 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5178 (secure 0 0 0) 0x7fa29c07ce50 con 0x7fa2a00d2360 2024-04-15T15:16:58.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:16:58.663 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:16:27.226372+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4907},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4907":{"gid":4907,"name":"d","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.96:6835/2334246723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":2334246723},{"type":"v1","addr":"172.21.15.96:6835","nonce":2334246723}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4907,"qdb_cluster":[4907]},"id":2}]} 2024-04-15T15:16:58.664 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 15 2024-04-15T15:16:58.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 msgr2=0x7fa28406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 0x7fa28406ee40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa290006f00 tx=0x7fa290008040 comp rx=0 tx=0).stop 2024-04-15T15:16:58.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 msgr2=0x7fa2a01f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa29c009de0 tx=0x7fa29c004a40 comp rx=0 tx=0).stop 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 shutdown_connections 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2a015eae0 0x7fa2a00b90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2a015e700 0x7fa2a00b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa28406c980 0x7fa28406ee40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 --2- 172.21.15.70:0/946563902 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2a00d2360 0x7fa2a01f7df0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 >> 172.21.15.70:0/946563902 conn(0x7fa2a008bf60 msgr2=0x7fa2a00d05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 shutdown_connections 2024-04-15T15:16:58.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.662+0000 7fa2a8750640 1 -- 172.21.15.70:0/946563902 wait complete. 2024-04-15T15:16:58.676 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb 0 2024-04-15T15:16:58.797 INFO:teuthology.orchestra.run.smithi096.stdout:[{"ino":1099511627779,"rdev":0,"ctime":"2024-04-15T15:16:58.095788+0000","btime":"2024-04-15T15:16:39.016390+0000","mode":16895,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":6,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2024-04-15T15:16:58.095788+0000","atime":"2024-04-15T15:16:39.016390+0000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"quiesce_block":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"2024-04-15T15:16:58.095788+0000","num_files":1,"num_subdirs":0,"change_attr":1},"rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:16:58.126912+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":1,"rsubdirs":1,"rsnaps":0,"rctime":"2024-04-15T15:16:58.126912+0000"},"version":20,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":1,"is_frozen":false,"is_freezing":false,"pins":{"request":1,"lock":2,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"waiter":0,"authpin":1},"nref":8,"versionlock":{"gather_set":[],"state":"lock","type":"iversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":1,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","type":"inest","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"quiescelock":{"gather_set":[],"state":"lock","type":"iquiesce","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":1,"xlock_by":{"reqid":{"entity":{"type":"mds","num":0},"tid":2}}},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":5126,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5461,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2},{"client_id":5778,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":7},{"client_id":5940,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":3},{"client_id":5964,"pending":"pAsLsXs","issued":"pAsLsXs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[],"dirfrags":[{"path":"/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb","dirfrag":"0x10000000003","snapid_first":2,"projected_version":"7","version":"7","committing_version":"0","committed_version":"0","is_rep":false,"dir_auth":"","states":["auth","dirty","complete"],"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"child":1,"dirty":1,"authpin":0},"nref":2,"dentries":[{"path":"volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/file","path_ino":1,"snap_first":2,"snap_last":18446744073709551614,"is_primary":true,"is_remote":false,"is_null":false,"is_new":false,"inode":1099511629282,"remote_type":"","version":6,"projected_version":6,"auth_pins":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"inodepin":1,"dirty":1,"authpin":0},"nref":2,"lock":{},"versionlock":{"gather_set":[],"state":"lock","type":"dversion","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"states":["auth","dirty"]}]}],"is_subvolume":false}] 2024-04-15T15:16:58.880 DEBUG:tasks.cephfs.filesystem:_json_asok output [ { "accounted_rstat": { "rbytes": 0, "rctime": "2024-04-15T15:16:58.126912+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2024-04-15T15:16:39.016390+0000", "auth_pins": 1, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 2, "btime": "2024-04-15T15:16:39.016390+0000", "change_attr": 2, "client_caps": [ { "client_id": 5126, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5461, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5778, "issued": "pAsLsXs", "last_sent": 7, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5940, "issued": "pAsLsXs", "last_sent": 3, "pending": "pAsLsXs", "wanted": "-" }, { "client_id": 5964, "issued": "pAsLsXs", "last_sent": 2, "pending": "pAsLsXs", "wanted": "-" } ], "client_ranges": [], "ctime": "2024-04-15T15:16:58.095788+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfrags": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "committed_version": "0", "committing_version": "0", "dentries": [ { "auth_pins": 0, "auth_state": { "replicas": {} }, "inode": 1099511629282, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_new": false, "is_null": false, "is_primary": true, "is_remote": false, "lock": {}, "nref": 2, "path": "volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/file", "path_ino": 1, "pins": { "authpin": 0, "dirty": 1, "inodepin": 1, "lock": 0, "request": 0 }, "projected_version": 6, "remote_type": "", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snap_first": 2, "snap_last": 18446744073709551614, "states": [ "auth", "dirty" ], "version": 6, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "dversion", "xlock_by": {} } } ], "dir_auth": "", "dirfrag": "0x10000000003", "is_auth": true, "is_freezing": false, "is_frozen": false, "is_rep": false, "nref": 2, "path": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb", "pins": { "authpin": 0, "child": 1, "dirty": 1 }, "projected_version": "7", "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "snapid_first": 2, "states": [ "auth", "dirty", "complete" ], "version": "7" } ], "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 1, "mtime": "2024-04-15T15:16:58.095788+0000", "num_files": 1, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1099511627779, "is_auth": true, "is_freezing": false, "is_frozen": false, "is_subvolume": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 6, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16895, "mtime": "2024-04-15T15:16:58.095788+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "type": "inest", "xlock_by": {} }, "nlink": 1, "nref": 8, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "pins": { "authpin": 1, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 2, "request": 1, "waiter": 0 }, "policylock": {}, "quiesce_block": false, "quiescelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "state": "lock", "type": "iquiesce", "xlock_by": { "reqid": { "entity": { "num": 0, "type": "mds" }, "tid": 2 } } }, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2024-04-15T15:16:58.126912+0000", "rfiles": 1, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 20, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "state": "lock", "type": "iversion", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } ] 2024-04-15T15:16:58.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.2:0 ops --flags=locks --path=/tmp/mds.0-ops 2024-04-15T15:16:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.984+0000 7f185c47e640 1 Processor -- start 2024-04-15T15:16:58.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.984+0000 7f185c47e640 1 -- start start 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.985+0000 7f185c47e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 0x7f1854160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.985+0000 7f185c47e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f185415fb30 0x7f18540d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.985+0000 7f185c47e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1854070720 con 0x7f18540701f0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.985+0000 7f185c47e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f185409ba90 con 0x7f185415fb30 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.985+0000 7f185c47e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f18540d08c0 con 0x7f185409a9d0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185a21a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 0x7f1854160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185aa1b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f185415fb30 0x7f18540d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f1859a19640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f18540701f0 0x7f185409adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:45210/0 (socket says 172.21.15.70:45210) 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f1859a19640 1 -- 172.21.15.70:0/2159491195 learned_addr learned my addr 172.21.15.70:0/2159491195 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f1859218640 1 -- 172.21.15.70:0/2159491195 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 423838414 0 0) 0x7f1854070720 con 0x7f18540701f0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f1859218640 1 -- 172.21.15.70:0/2159491195 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f183c003520 con 0x7f18540701f0 2024-04-15T15:16:58.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185aa1b640 1 -- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 msgr2=0x7f1854160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185aa1b640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 0x7f1854160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185aa1b640 1 -- 172.21.15.70:0/2159491195 >> v1:172.21.15.96:6789/0 conn(0x7f18540701f0 legacy=0x7f185409adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185aa1b640 1 -- 172.21.15.70:0/2159491195 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1854070a90 con 0x7f185415fb30 2024-04-15T15:16:58.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.986+0000 7f185a21a640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 0x7f1854160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:58.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f185aa1b640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185415fb30 0x7f18540d0d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f184c0097d0 tx=0x7f184c02b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fb67b160bbc7a30 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f1859218640 1 -- 172.21.15.70:0/2159491195 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c035020 con 0x7f185415fb30 2024-04-15T15:16:58.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f1859218640 1 -- 172.21.15.70:0/2159491195 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c031800 con 0x7f185415fb30 2024-04-15T15:16:58.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f1859218640 1 -- 172.21.15.70:0/2159491195 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c031bb0 con 0x7f185415fb30 2024-04-15T15:16:58.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f185c47e640 1 -- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185415fb30 msgr2=0x7f18540d0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.987+0000 7f185c47e640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185415fb30 0x7f18540d0d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f184c0097d0 tx=0x7f184c02b8a0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 -- 172.21.15.70:0/2159491195 shutdown_connections 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185415fb30 0x7f18540d0d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 --2- 172.21.15.70:0/2159491195 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f185409a9d0 0x7f1854160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 -- 172.21.15.70:0/2159491195 >> 172.21.15.70:0/2159491195 conn(0x7f18540d2460 msgr2=0x7f18540d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 -- 172.21.15.70:0/2159491195 shutdown_connections 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.988+0000 7f185c47e640 1 -- 172.21.15.70:0/2159491195 wait complete. 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 Processor -- start 2024-04-15T15:16:58.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 -- start start 2024-04-15T15:16:58.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f18540701f0 0x7f18540ce860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185409a9d0 0x7f18540b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:58.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f185415ffc0 con 0x7f185415fb30 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1854071050 con 0x7f185409a9d0 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.989+0000 7f185c47e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f18540b1c60 con 0x7f18540701f0 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46794/0 (socket says 172.21.15.70:46794) 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 -- 172.21.15.70:0/3429183101 learned_addr learned my addr 172.21.15.70:0/3429183101 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185a21a640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f18540701f0 0x7f18540ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f1859a19640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185409a9d0 0x7f18540b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 -- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f18540701f0 msgr2=0x7f18540ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f18540701f0 0x7f18540ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 -- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185409a9d0 msgr2=0x7f18540b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185409a9d0 0x7f18540b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 -- 172.21.15.70:0/3429183101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18540ce500 con 0x7f185415fb30 2024-04-15T15:16:58.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185aa1b640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f184c02beb0 tx=0x7f184c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f1852ffd640 1 -- 172.21.15.70:0/3429183101 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c035040 con 0x7f185415fb30 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18540c8790 con 0x7f185415fb30 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f1852ffd640 1 -- 172.21.15.70:0/3429183101 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c03ad00 con 0x7f185415fb30 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f1852ffd640 1 -- 172.21.15.70:0/3429183101 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c006e10 con 0x7f185415fb30 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.990+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18540c8d50 con 0x7f185415fb30 2024-04-15T15:16:58.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.992+0000 7f1852ffd640 1 -- 172.21.15.70:0/3429183101 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f184c038020 con 0x7f185415fb30 2024-04-15T15:16:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.992+0000 7f1852ffd640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 0x7f184006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.992+0000 7f1852ffd640 1 -- 172.21.15.70:0/3429183101 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f184c0add10 con 0x7f185415fb30 2024-04-15T15:16:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.992+0000 7f185a21a640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 0x7f184006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.993+0000 7f185a21a640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 0x7f184006ee40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1844009bd0 tx=0x7f184400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.993+0000 7f185f061740 1 Processor -- start 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.993+0000 7f185f061740 1 -- start start 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858a8da0 0x563f858a9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858acb60 0x563f858ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563f85813a80 con 0x563f858d88a0 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563f857ba580 con 0x563f858acb60 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x563f8580fa40 con 0x563f858a8da0 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46798/0 (socket says 172.21.15.70:46798) 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 -- 172.21.15.70:0/2415223305 learned_addr learned my addr 172.21.15.70:0/2415223305 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f1859a19640 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858acb60 0x563f858ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185a21a640 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858a8da0 0x563f858a9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 -- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858a8da0 msgr2=0x563f858a9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858a8da0 0x563f858a9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 -- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858acb60 msgr2=0x563f858ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858acb60 0x563f858ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.994+0000 7f185aa1b640 1 -- 172.21.15.70:0/2415223305 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f857cead0 con 0x563f858d88a0 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f185aa1b640 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f184c0ac050 tx=0x7f184c07b230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33bb4ada4b722b06 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f183bfff640 1 -- 172.21.15.70:0/2415223305 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c04b070 con 0x563f858d88a0 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f183bfff640 1 -- 172.21.15.70:0/2415223305 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c084400 con 0x563f858d88a0 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f183bfff640 1 -- 172.21.15.70:0/2415223305 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c077f40 con 0x563f858d88a0 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f185f061740 1 -- 172.21.15.70:0/2415223305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 msgr2=0x563f858d8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.995+0000 7f185f061740 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f184c0ac050 tx=0x7f184c07b230 comp rx=0 tx=0).stop 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 -- 172.21.15.70:0/2415223305 shutdown_connections 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858d8d20 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858acb60 0x563f858ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 --2- 172.21.15.70:0/2415223305 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858a8da0 0x563f858a9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 -- 172.21.15.70:0/2415223305 >> 172.21.15.70:0/2415223305 conn(0x563f858aae20 msgr2=0x563f858ac450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 -- 172.21.15.70:0/2415223305 shutdown_connections 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 -- 172.21.15.70:0/2415223305 wait complete. 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 -- start start 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.996+0000 7f185f061740 1 Processor -- start 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858d31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 0x563f858d4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563f858acff0 con 0x563f858cd3f0 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x563f857cd940 con 0x563f858cb8d0 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x563f858ad2d0 con 0x563f858d88a0 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185f061740 10 client.0 _ll_register_callbacks cb 0x563f857cd350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f1859a19640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f1859a19640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46804/0 (socket says 172.21.15.70:46804) 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f1859a19640 1 -- 172.21.15.70:0/4021817079 learned_addr learned my addr 172.21.15.70:0/4021817079 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:59.007 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185a21a640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858d31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.997+0000 7f185aa1b640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 0x563f858d4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 -- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 msgr2=0x563f858d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858d31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 -- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 msgr2=0x563f858d4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 0x563f858d4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 -- 172.21.15.70:0/4021817079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f85526420 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185aa1b640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 0x563f858d4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f1859a19640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1848000f60 tx=0x7f1848012690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73b86653cfac4f3a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1848019020 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f18480058b0 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f184c084400 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f1848023050 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 10 client.6012 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x563f855f2830 con 0x563f858cd3f0 2024-04-15T15:16:59.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c084400 con 0x563f858cd3f0 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f1848026020 con 0x563f858cd3f0 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 10 client.6012 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 10 client.6012 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 20 client.6012 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858d4e40 0x563f858d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 4 client.6012 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.998+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f858214a0 con 0x563f858d4e40 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f185a21a640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858d4e40 0x563f858d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f18397fa640 10 client.6012 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f185a21a640 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858d4e40 0x563f858d0df0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=199bdbf0289a00f1 server_cookie=b71fbc00ac63a452 in_seq=0 out_seq=0 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f18397fa640 1 -- 172.21.15.70:0/4021817079 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f858214a0 con 0x563f858d4e40 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f18397fa640 10 client.6012 handle_command_reply: tid=0 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:58.999+0000 7f185f061740 1 client.6012 shutdown 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 msgr2=0x563f858d3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1848000f60 tx=0x7f1848012690 comp rx=0 tx=0).stop 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 shutdown_connections 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cb8d0 0x563f858d4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cd3f0 0x563f858d3710 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858d4e40 0x563f858d0df0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 --2- 172.21.15.70:0/4021817079 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858d31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 >> 172.21.15.70:0/4021817079 conn(0x563f858a8da0 msgr2=0x563f858d6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f18397fa640 0 client.6012 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 shutdown_connections 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 1 -- 172.21.15.70:0/4021817079 wait complete. 2024-04-15T15:16:59.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.000+0000 7f185f061740 20 client.6012 trim_cache size 0 max 16384 2024-04-15T15:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 Processor -- start 2024-04-15T15:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 -- start start 2024-04-15T15:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f858d6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858a7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f18542011f0 con 0x563f858cb8d0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f185406d790 con 0x563f858cd3f0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f185406caf0 con 0x563f858d88a0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.016+0000 7f185a21a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46820/0 (socket says 172.21.15.70:46820) 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 -- 172.21.15.70:0/1642526743 learned_addr learned my addr 172.21.15.70:0/1642526743 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f1859a19640 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f858d6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185aa1b640 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 -- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 msgr2=0x563f858a7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858a7980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 -- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 msgr2=0x563f858d6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f858d6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 -- 172.21.15.70:0/1642526743 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f858d47a0 con 0x563f858cb8d0 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f185a21a640 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x563f858d53a0 tx=0x7f1844014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44508575f39342c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f183b7fe640 1 -- 172.21.15.70:0/1642526743 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184402cdc0 con 0x563f858cb8d0 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f183b7fe640 1 -- 172.21.15.70:0/1642526743 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844031070 con 0x563f858cb8d0 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.017+0000 7f183b7fe640 1 -- 172.21.15.70:0/1642526743 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184402c580 con 0x563f858cb8d0 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- 172.21.15.70:0/1642526743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 msgr2=0x563f858f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x563f858d53a0 tx=0x7f1844014000 comp rx=0 tx=0).stop 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- 172.21.15.70:0/1642526743 shutdown_connections 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858d88a0 0x563f858a7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f858d6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 --2- 172.21.15.70:0/1642526743 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858cb8d0 0x563f858f6d70 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- 172.21.15.70:0/1642526743 >> 172.21.15.70:0/1642526743 conn(0x563f858a8da0 msgr2=0x563f858dc110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- 172.21.15.70:0/1642526743 shutdown_connections 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- 172.21.15.70:0/1642526743 wait complete. 2024-04-15T15:16:59.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 -- start start 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.018+0000 7f185f061740 1 Processor -- start 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858cb8d0 0x563f859638f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f85965b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x563f858d3cb0 con 0x563f858d88a0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f185406d140 con 0x563f858cd3f0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x563f858cb690 con 0x563f858cb8d0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185f061740 10 client.0 _ll_register_callbacks cb 0x7f1848001b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46826/0 (socket says 172.21.15.70:46826) 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 -- 172.21.15.70:0/649442697 learned_addr learned my addr 172.21.15.70:0/649442697 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f1859a19640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f85965b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185a21a640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858cb8d0 0x563f859638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 -- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858cb8d0 msgr2=0x563f859638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858cb8d0 0x563f859638f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 -- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 msgr2=0x563f85965b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f85965b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.019+0000 7f185aa1b640 1 -- 172.21.15.70:0/649442697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f859181b0 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185aa1b640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f184c07b860 tx=0x7f184c04b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1da9ad1466dbf47 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c084400 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f184c07ad20 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1844031070 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f184c0a2050 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 10 client.6018 fetch_fsmap learned FSMap version 15 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x563f85963420 con 0x563f858d88a0 2024-04-15T15:16:59.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1844031070 con 0x563f858d88a0 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f184c0a5020 con 0x563f858d88a0 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 10 client.6018 fetch_fsmap finished waiting for FSMap version 15 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 10 client.6018 resolve_mds: resolved 2:0 to role '2:0' aka daemon mds.d 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 20 client.6018 populate_metadata read hostname 'smithi070' 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858f7280 0x563f858cec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 4 client.6018 mds_command: new command op to 4907 tid=0 multi_id=0 [{"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}] 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 --> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] -- command(tid 0: {"prefix": "ops", "flags": ["locks"], "path": "/tmp/mds.0-ops"}) -- 0x563f858ab580 con 0x563f858f7280 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.020+0000 7f185a21a640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858f7280 0x563f858cec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f18397fa640 10 client.6018 ms_handle_connect on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f185a21a640 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858f7280 0x563f858cec20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4055bb7a77b8e7da server_cookie=de2b4fe2e269b4ff in_seq=0 out_seq=0 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f18397fa640 1 -- 172.21.15.70:0/649442697 <== mds.0 v2:172.21.15.96:6833/2334246723 1 ==== command_reply(tid 0: 0 ) ==== 8+0+33 (crc 0 0 0) 0x563f858ab580 con 0x563f858f7280 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f18397fa640 10 client.6018 handle_command_reply: tid=0 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f185f061740 1 client.6018 shutdown 2024-04-15T15:16:59.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 msgr2=0x563f858ce6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.021+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f184c07b860 tx=0x7f184c04b000 comp rx=0 tx=0).stop 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 shutdown_connections 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x563f858d88a0 0x563f858ce6e0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x563f858cd3f0 0x563f85965b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:6833/2334246723,v1:172.21.15.96:6835/2334246723] conn(0x563f858f7280 0x563f858cec20 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 --2- 172.21.15.70:0/649442697 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x563f858cb8d0 0x563f859638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 >> 172.21.15.70:0/649442697 conn(0x563f858a8da0 msgr2=0x563f858a7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f18397fa640 0 client.6018 ms_handle_reset on v2:172.21.15.96:6833/2334246723 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 shutdown_connections 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 1 -- 172.21.15.70:0/649442697 wait complete. 2024-04-15T15:16:59.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185f061740 20 client.6018 trim_cache size 0 max 16384 2024-04-15T15:16:59.032 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:16:59.032 INFO:teuthology.orchestra.run.smithi070.stdout: "path": "/tmp/mds.0-ops" 2024-04-15T15:16:59.032 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 msgr2=0x7f184006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 0x7f184006ee40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f1844009bd0 tx=0x7f184400b040 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 msgr2=0x7f18540cdfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f184c02beb0 tx=0x7f184c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 shutdown_connections 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f184006c980 0x7f184006ee40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f185415fb30 0x7f18540cdfc0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f185409a9d0 0x7f18540b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 --2- 172.21.15.70:0/3429183101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f18540701f0 0x7f18540ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 >> 172.21.15.70:0/3429183101 conn(0x7f18540d2460 msgr2=0x7f185415e940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 shutdown_connections 2024-04-15T15:16:59.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.022+0000 7f185c47e640 1 -- 172.21.15.70:0/3429183101 wait complete. 2024-04-15T15:16:59.036 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-04-15T15:16:59.051 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.Sqrina8bu9 2024-04-15T15:16:59.052 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /tmp/mds.0-ops /tmp/tmp.Sqrina8bu9 2024-04-15T15:16:59.121 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.Sqrina8bu9 2024-04-15T15:16:59.249 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.Sqrina8bu9 is 8KB 2024-04-15T15:16:59.298 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.Sqrina8bu9 2024-04-15T15:16:59.313 DEBUG:tasks.cephfs.filesystem:read 7749B of ops 2024-04-15T15:16:59.314 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json dump tree /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb --path=/tmp/mds.0-cache 2024-04-15T15:16:59.472 INFO:teuthology.orchestra.run.smithi096.stdout:{"path":"/tmp/mds.0-cache"} 2024-04-15T15:16:59.472 DEBUG:tasks.cephfs.filesystem:_json_asok output { "path": "/tmp/mds.0-cache" } 2024-04-15T15:16:59.472 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-04-15T15:16:59.488 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.igfKI0elAi 2024-04-15T15:16:59.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /tmp/mds.0-cache /tmp/tmp.igfKI0elAi 2024-04-15T15:16:59.558 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.igfKI0elAi 2024-04-15T15:16:59.686 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.igfKI0elAi is 6KB 2024-04-15T15:16:59.733 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.igfKI0elAi 2024-04-15T15:16:59.748 DEBUG:tasks.cephfs.filesystem:read 6360B of cache 2024-04-15T15:16:59.749 INFO:tasks.cephfs.test_quiesce:qp = 1; qi = 1; qib = 1; qina = 0 2024-04-15T15:16:59.749 ERROR:tasks.cephfs.test_quiesce:inode: { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:16:58.095788+0000", "btime": "2024-04-15T15:16:39.016390+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:58.095788+0000", "atime": "2024-04-15T15:16:39.016390+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:16:58.095788+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5126, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5461, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5778, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5940, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5964, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } 2024-04-15T15:16:59.750 ERROR:tasks.cephfs.test_quiesce:op: { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:16:58.317759+0000", "age": 0.705732677, "duration": 0.705859179, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:58.317759+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:58.317935+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:58.318389+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:58.320893+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:58.095788+0000 1=1+0) n(v0 rc2024-04-15T15:16:58.126912+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5126=pAsLsXs/-@2,5461=pAsLsXs/-@2,5778=pAsLsXs/-@7,5940=pAsLsXs/-@3,5964=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x55ab1b22b080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:58.095788+0000 1=1+0) n(v0 rc2024-04-15T15:16:58.126912+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5126=pAsLsXs/-@2,5461=pAsLsXs/-@2,5778=pAsLsXs/-@7,5940=pAsLsXs/-@3,5964=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x55ab1b22b080]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } } 2024-04-15T15:16:59.750 ERROR:tasks.cephfs.test_quiesce:lock_type: iversion 2024-04-15T15:16:59.750 ERROR:tasks.cephfs.test_quiesce:cache: [ { "ino": 1099511629282, "rdev": 0, "ctime": "2024-04-15T15:16:58.126912+0000", "btime": "2024-04-15T15:16:58.095788+0000", "mode": 33188, "uid": 1000, "gid": 1301, "nlink": 1, "dir_layout": { "dir_hash": 0, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:58.101996+0000", "atime": "2024-04-15T15:16:58.101996+0000", "time_warp_seq": 2, "change_attr": 1, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": true, "client_ranges": [], "dirstat": { "version": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "change_attr": 0 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 0, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "version": 6, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 4, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "ptrwaiter": 0, "request": 1, "lock": 0, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 5, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5940, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [], "is_subvolume": false }, { "ino": 1099511627779, "rdev": 0, "ctime": "2024-04-15T15:16:58.095788+0000", "btime": "2024-04-15T15:16:39.016390+0000", "mode": 16895, "uid": 0, "gid": 0, "nlink": 1, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "layout": { "stripe_unit": 4194304, "stripe_count": 1, "object_size": 4194304, "pool_id": 6, "pool_ns": "" }, "old_pools": [], "size": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "truncate_from": 0, "truncate_pending": 0, "mtime": "2024-04-15T15:16:58.095788+0000", "atime": "2024-04-15T15:16:39.016390+0000", "time_warp_seq": 0, "change_attr": 2, "export_pin": -1, "export_ephemeral_random_pin": 0, "export_ephemeral_distributed_pin": false, "quiesce_block": false, "client_ranges": [], "dirstat": { "version": 0, "mtime": "2024-04-15T15:16:58.095788+0000", "num_files": 1, "num_subdirs": 0, "change_attr": 1 }, "rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "accounted_rstat": { "version": 0, "rbytes": 0, "rfiles": 1, "rsubdirs": 1, "rsnaps": 0, "rctime": "2024-04-15T15:16:58.126912+0000" }, "version": 20, "file_data_version": 0, "xattr_version": 1, "backtrace_version": 2, "stray_prior_path": "", "max_size_ever": 0, "quota": { "max_bytes": 0, "max_files": 0 }, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "symlink": "", "xattrs": [], "dirfragtree": { "splits": [] }, "old_inodes": [], "oldest_snap": 18446744073709551614, "damage_flags": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8, "versionlock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "authlock": {}, "linklock": {}, "dirfragtreelock": {}, "filelock": {}, "xattrlock": {}, "snaplock": {}, "nestlock": { "gather_set": [], "state": "lock", "type": "inest", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "flocklock": {}, "policylock": {}, "quiescelock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "client_caps": [ { "client_id": 5126, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5461, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 }, { "client_id": 5778, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 7 }, { "client_id": 5940, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 3 }, { "client_id": 5964, "pending": "pAsLsXs", "issued": "pAsLsXs", "wanted": "-", "last_sent": 2 } ], "loner": -1, "want_loner": -1, "mds_caps_wanted": [], "dirfrags": [ { "path": "/volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb", "dirfrag": "0x10000000003", "snapid_first": 2, "projected_version": "7", "version": "7", "committing_version": "0", "committed_version": "0", "is_rep": false, "dir_auth": "", "states": [ "auth", "dirty", "complete" ], "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 0, "is_frozen": false, "is_freezing": false, "pins": { "child": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "dentries": [ { "path": "volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/file", "path_ino": 1, "snap_first": 2, "snap_last": 18446744073709551614, "is_primary": true, "is_remote": false, "is_null": false, "is_new": false, "inode": 1099511629282, "remote_type": "", "version": 6, "projected_version": 6, "auth_pins": 0, "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "is_frozen": false, "is_freezing": false, "pins": { "request": 0, "lock": 0, "inodepin": 1, "dirty": 1, "authpin": 0 }, "nref": 2, "lock": {}, "versionlock": { "gather_set": [], "state": "lock", "type": "dversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "xlock_by": {} }, "states": [ "auth", "dirty" ] } ] } ], "is_subvolume": false } ] 2024-04-15T15:16:59.751 ERROR:tasks.cephfs.test_quiesce:ops: { "1099511627779": { "description": "internal op quiesce_inode:mds.0:2 fp=#0x10000000003", "initiated_at": "2024-04-15T15:16:58.317759+0000", "age": 0.705732677, "duration": 0.705859179, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce complete", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:58.317759+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:58.317759+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:58.317935+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:58.318389+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:58.320893+0000", "event": "quiesce complete" } ], "locks": [ { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:58.095788+0000 1=1+0) n(v0 rc2024-04-15T15:16:58.126912+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5126=pAsLsXs/-@2,5461=pAsLsXs/-@2,5778=pAsLsXs/-@7,5940=pAsLsXs/-@3,5964=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x55ab1b22b080]", "lock": { "gather_set": [], "state": "lock", "type": "iquiesce", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 1, "xlock_by": { "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 2 } } }, "flags": 4, "wrlock_target": -1 }, { "object": { "is_auth": true, "auth_state": { "replicas": {} }, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "auth_pins": 1, "is_frozen": false, "is_freezing": false, "pins": { "request": 1, "lock": 2, "dirfrag": 1, "caps": 1, "dirtyparent": 1, "dirty": 1, "waiter": 0, "authpin": 1 }, "nref": 8 }, "object_string": "[inode 0x10000000003 [...2,head] /volumes/_nogroup/subvol_quiesce/f4d2ab5b-9035-4cee-a372-737854ea4adb/ auth v20 ap=1 DIRTYPARENT f(v0 m2024-04-15T15:16:58.095788+0000 1=1+0) n(v0 rc2024-04-15T15:16:58.126912+0000 2=1+1) (inest lock) (iversion lock w=1) (iquiesce lock x=1 by request(mds.0:2 nref=3)) caps={5126=pAsLsXs/-@2,5461=pAsLsXs/-@2,5778=pAsLsXs/-@7,5940=pAsLsXs/-@3,5964=pAsLsXs/-@2} | request=1 lock=2 dirfrag=1 caps=1 dirtyparent=1 dirty=1 waiter=0 authpin=1 0x55ab1b22b080]", "lock": { "gather_set": [], "state": "lock", "type": "iversion", "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 1, "num_xlocks": 0, "xlock_by": {} }, "flags": 2, "wrlock_target": -1 } ] } }, "1099511629282": { "description": "internal op quiesce_inode:mds.0:3 fp=#0x100000005e2", "initiated_at": "2024-04-15T15:16:58.318526+0000", "age": 0.704965455, "duration": 0.705178745, "continuous": true, "type_data": { "result": -2147483648, "flag_point": "quiesce blocked", "reqid": { "entity": { "type": "mds", "num": 0 }, "tid": 3 }, "op_type": "internal_op", "internal_op": 5385, "op_name": "quiesce_inode", "events": [ { "time": "2024-04-15T15:16:58.318526+0000", "event": "initiated" }, { "time": "2024-04-15T15:16:58.318526+0000", "event": "header_read" }, { "time": "2024-04-15T15:16:58.318526+0000", "event": "throttled" }, { "time": "2024-04-15T15:16:58.318526+0000", "event": "all_read" }, { "time": "2024-04-15T15:16:58.318526+0000", "event": "dispatched" }, { "time": "2024-04-15T15:16:58.318637+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:58.319242+0000", "event": "failed to xlock, waiting" }, { "time": "2024-04-15T15:16:58.320765+0000", "event": "acquired locks" }, { "time": "2024-04-15T15:16:58.320767+0000", "event": "quiesce blocked" } ], "locks": [] } } } 2024-04-15T15:16:59.751 INFO:tasks.cephfs.test_quiesce:making archive directory /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/quiesce 2024-04-15T15:16:59.753 ERROR:tasks.cephfs.test_quiesce:cache written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/quiesce/cachegtf2ciwu 2024-04-15T15:16:59.755 ERROR:tasks.cephfs.test_quiesce:ops written to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/quiesce/opspszu2wwj 2024-04-15T15:16:59.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:16:59.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.858+0000 7f005c2c8640 1 Processor -- start 2024-04-15T15:16:59.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.858+0000 7f005c2c8640 1 -- start start 2024-04-15T15:16:59.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005c2c8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0054071d10 0x7f00540d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005c2c8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f00540d0bd0 0x7f005406e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005c2c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f005415db40 con 0x7f0054072d80 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005c2c8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f00540d1040 con 0x7f0054071d10 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005c2c8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0054072170 con 0x7f00540d0bd0 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059863640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0054071d10 0x7f00540d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005a064640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0054072d80 0x7f00540af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34744/0 (socket says 172.21.15.70:34744) 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f005a064640 1 -- 172.21.15.70:0/1460641542 learned_addr learned my addr 172.21.15.70:0/1460641542 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059863640 1 -- 172.21.15.70:0/1460641542 >> v2:172.21.15.96:3300/0 conn(0x7f00540d0bd0 msgr2=0x7f005406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059863640 1 --2- 172.21.15.70:0/1460641542 >> v2:172.21.15.96:3300/0 conn(0x7f00540d0bd0 0x7f005406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059863640 1 -- 172.21.15.70:0/1460641542 >> v1:172.21.15.70:6789/0 conn(0x7f0054072d80 legacy=0x7f00540af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059062640 1 -- 172.21.15.70:0/1460641542 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3101255099 0 0) 0x7f005415db40 con 0x7f0054072d80 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.859+0000 7f0059863640 1 -- 172.21.15.70:0/1460641542 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005415cfc0 con 0x7f0054071d10 2024-04-15T15:16:59.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.860+0000 7f0059863640 1 --2- 172.21.15.70:0/1460641542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540d0690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f00540b54c0 tx=0x7f004402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66175517734b5042 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.860+0000 7f0059062640 1 -- 172.21.15.70:0/1460641542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00440315a0 con 0x7f0054071d10 2024-04-15T15:16:59.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.860+0000 7f0059062640 1 -- 172.21.15.70:0/1460641542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0044037070 con 0x7f0054071d10 2024-04-15T15:16:59.863 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.860+0000 7f0059062640 1 -- 172.21.15.70:0/1460641542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0044031960 con 0x7f0054071d10 2024-04-15T15:16:59.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 -- 172.21.15.70:0/1460641542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 msgr2=0x7f00540d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1460641542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540d0690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f00540b54c0 tx=0x7f004402b840 comp rx=0 tx=0).stop 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 -- 172.21.15.70:0/1460641542 shutdown_connections 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1460641542 >> v2:172.21.15.96:3300/0 conn(0x7f00540d0bd0 0x7f005406e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1460641542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540d0690 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 -- 172.21.15.70:0/1460641542 >> 172.21.15.70:0/1460641542 conn(0x7f005406be00 msgr2=0x7f005406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 -- 172.21.15.70:0/1460641542 shutdown_connections 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.861+0000 7f005c2c8640 1 -- 172.21.15.70:0/1460641542 wait complete. 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.862+0000 7f005c2c8640 1 Processor -- start 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.862+0000 7f005c2c8640 1 -- start start 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.862+0000 7f005c2c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.862+0000 7f005c2c8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0054072d80 0x7f0054159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00540d0bd0 0x7f00540b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f005415d860 con 0x7f0054071d10 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f005415d580 con 0x7f00540d0bd0 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f00540afaf0 con 0x7f0054072d80 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46836/0 (socket says 172.21.15.70:46836) 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 -- 172.21.15.70:0/1332041855 learned_addr learned my addr 172.21.15.70:0/1332041855 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 -- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0054072d80 msgr2=0x7f0054159790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0054072d80 0x7f0054159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a865640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00540d0bd0 0x7f00540b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 -- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00540d0bd0 msgr2=0x7f00540b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00540d0bd0 0x7f00540b41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 -- 172.21.15.70:0/1332041855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f005414ea60 con 0x7f0054071d10 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005a064640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f005000c4d0 tx=0x7f005000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050018070 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0050007500 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00540ccfd0 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050004a20 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.863+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00540cd870 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.864+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00540689e0 con 0x7f0054071d10 2024-04-15T15:16:59.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.865+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f005001b070 con 0x7f0054071d10 2024-04-15T15:16:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.865+0000 7f004affd640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 0x7f003806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:16:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.865+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6130+0+0 (secure 0 0 0) 0x7f005008b050 con 0x7f0054071d10 2024-04-15T15:16:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.865+0000 7f0059863640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 0x7f003806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:16:59.869 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.866+0000 7f0059863640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 0x7f003806ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0044036000 tx=0x7f0044030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:16:59.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:16:59.870+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f00500592e0 con 0x7f0054071d10 2024-04-15T15:17:00.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.082+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f00540b2850 con 0x7f0054071d10 2024-04-15T15:17:00.084 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-15T15:17:00.083+0000 7fdd5b7bd640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:17:00.085 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.084+0000 7f004affd640 1 -- 172.21.15.70:0/1332041855 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) ==== 161+0+0 (secure 0 0 0) 0x7f0050058be0 con 0x7f0054071d10 2024-04-15T15:17:00.085 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:17:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 msgr2=0x7f003806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 0x7f003806ee40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0044036000 tx=0x7f0044030000 comp rx=0 tx=0).stop 2024-04-15T15:17:00.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 msgr2=0x7f00540c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f005000c4d0 tx=0x7f005000c990 comp rx=0 tx=0).stop 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 shutdown_connections 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f00540d0bd0 0x7f00540b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0054072d80 0x7f0054159790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f003806c980 0x7f003806ee40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 --2- 172.21.15.70:0/1332041855 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0054071d10 0x7f00540c9a80 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 >> 172.21.15.70:0/1332041855 conn(0x7f005406be00 msgr2=0x7f0054074d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 shutdown_connections 2024-04-15T15:17:00.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.086+0000 7f005c2c8640 1 -- 172.21.15.70:0/1332041855 wait complete. 2024-04-15T15:17:00.100 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-15T15:17:00.126 INFO:tasks.ceph.mds.d.smithi096.stdout:starting mds.d at 2024-04-15T15:17:00.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.204+0000 7f2e61549640 1 Processor -- start 2024-04-15T15:17:00.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.204+0000 7f2e61549640 1 -- start start 2024-04-15T15:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e61549640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e5c15e700 0x7f2e5c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e61549640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 0x7f2e5c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e61549640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e5c0b47a0 con 0x7f2e5c0d2360 2024-04-15T15:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e61549640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f2e5c0b53b0 con 0x7f2e5c15e700 2024-04-15T15:17:00.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e61549640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f2e5c0b4a80 con 0x7f2e5c0d3270 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e5affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f2e5c15e700 0x7f2e5c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e5a7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2e5c0d2360 0x7f2e5c0d2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34760/0 (socket says 172.21.15.70:34760) 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.205+0000 7f2e5a7fc640 1 -- 172.21.15.70:0/4225587136 learned_addr learned my addr 172.21.15.70:0/4225587136 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5b7fe640 1 --2- 172.21.15.70:0/4225587136 >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 0x7f2e5c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5affd640 1 -- 172.21.15.70:0/4225587136 >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 msgr2=0x7f2e5c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5affd640 1 --2- 172.21.15.70:0/4225587136 >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 0x7f2e5c0cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5affd640 1 -- 172.21.15.70:0/4225587136 >> v1:172.21.15.70:6789/0 conn(0x7f2e5c0d2360 legacy=0x7f2e5c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5affd640 1 -- 172.21.15.70:0/4225587136 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c0b4d60 con 0x7f2e5c15e700 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e59ffb640 1 -- 172.21.15.70:0/4225587136 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4161865880 0 0) 0x7f2e5c0b47a0 con 0x7f2e5c0d2360 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.206+0000 7f2e5b7fe640 1 --2- 172.21.15.70:0/4225587136 >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 0x7f2e5c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e5affd640 1 --2- 172.21.15.70:0/4225587136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c15e700 0x7f2e5c0b5840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0d1b00 tx=0x7f2e4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe121b71b54b1bb4 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e59ffb640 1 -- 172.21.15.70:0/4225587136 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c035020 con 0x7f2e5c15e700 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e59ffb640 1 -- 172.21.15.70:0/4225587136 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e4c031830 con 0x7f2e5c15e700 2024-04-15T15:17:00.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e59ffb640 1 -- 172.21.15.70:0/4225587136 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c031c00 con 0x7f2e5c15e700 2024-04-15T15:17:00.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 -- 172.21.15.70:0/4225587136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c15e700 msgr2=0x7f2e5c0b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 --2- 172.21.15.70:0/4225587136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c15e700 0x7f2e5c0b5840 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0d1b00 tx=0x7f2e4c02b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 -- 172.21.15.70:0/4225587136 shutdown_connections 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 --2- 172.21.15.70:0/4225587136 >> v2:172.21.15.96:3300/0 conn(0x7f2e5c0d3270 0x7f2e5c0cf370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 --2- 172.21.15.70:0/4225587136 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c15e700 0x7f2e5c0b5840 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 -- 172.21.15.70:0/4225587136 >> 172.21.15.70:0/4225587136 conn(0x7f2e5c08bf60 msgr2=0x7f2e5c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 -- 172.21.15.70:0/4225587136 shutdown_connections 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.207+0000 7f2e61549640 1 -- 172.21.15.70:0/4225587136 wait complete. 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 Processor -- start 2024-04-15T15:17:00.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 -- start start 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2e5c0d2360 0x7f2e5c1615c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e5c15e700 0x7f2e5c1f7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c0b5d80 con 0x7f2e5c0d3270 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f2e5c0b4450 con 0x7f2e5c15e700 2024-04-15T15:17:00.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e61549640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f2e5c0d1d40 con 0x7f2e5c0d2360 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e5a7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46854/0 (socket says 172.21.15.70:46854) 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.209+0000 7f2e5a7fc640 1 -- 172.21.15.70:0/3920415470 learned_addr learned my addr 172.21.15.70:0/3920415470 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5affd640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2e5c0d2360 0x7f2e5c1615c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5b7fe640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e5c15e700 0x7f2e5c1f7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 -- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2e5c0d2360 msgr2=0x7f2e5c1615c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2e5c0d2360 0x7f2e5c1615c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 -- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e5c15e700 msgr2=0x7f2e5c1f7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e5c15e700 0x7f2e5c1f7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 -- 172.21.15.70:0/3920415470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e5c0b4240 con 0x7f2e5c0d3270 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e5a7fc640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000c500 tx=0x7f2e5000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50018070 con 0x7f2e5c0d3270 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e50007580 con 0x7f2e5c0d3270 2024-04-15T15:17:00.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e50004a20 con 0x7f2e5c0d3270 2024-04-15T15:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e5c0b8440 con 0x7f2e5c0d3270 2024-04-15T15:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.210+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e5c0b9460 con 0x7f2e5c0d3270 2024-04-15T15:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.211+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e5c0689e0 con 0x7f2e5c0d3270 2024-04-15T15:17:00.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.212+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2e5001b070 con 0x7f2e5c0d3270 2024-04-15T15:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.212+0000 7f2e437fe640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 0x7f2e4806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.212+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f2e5008af10 con 0x7f2e5c0d3270 2024-04-15T15:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.212+0000 7f2e5affd640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 0x7f2e4806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.213+0000 7f2e5affd640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 0x7f2e4806ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c036000 tx=0x7f2e4c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.218 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.216+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2e50058b90 con 0x7f2e5c0d3270 2024-04-15T15:17:00.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.429+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f2e5c0ba500 con 0x7f2e5c0d3270 2024-04-15T15:17:00.632 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.630+0000 7f2e437fe640 1 -- 172.21.15.70:0/3920415470 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v17) ==== 162+0+0 (secure 0 0 0) 0x7f2e5005ce90 con 0x7f2e5c0d3270 2024-04-15T15:17:00.632 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 msgr2=0x7f2e4806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 0x7f2e4806ee40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c036000 tx=0x7f2e4c03c040 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 msgr2=0x7f2e5c0b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000c500 tx=0x7f2e5000c9c0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 shutdown_connections 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f2e5c15e700 0x7f2e5c1f7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f2e4806c980 0x7f2e4806ee40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e5c0d3270 0x7f2e5c0b3d00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 --2- 172.21.15.70:0/3920415470 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f2e5c0d2360 0x7f2e5c1615c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 >> 172.21.15.70:0/3920415470 conn(0x7f2e5c08bf60 msgr2=0x7f2e5c0cfbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 shutdown_connections 2024-04-15T15:17:00.635 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.633+0000 7f2e61549640 1 -- 172.21.15.70:0/3920415470 wait complete. 2024-04-15T15:17:00.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:00.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.750+0000 7f7624695640 1 Processor -- start 2024-04-15T15:17:00.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.750+0000 7f7624695640 1 -- start start 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.751+0000 7f7624695640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f761c0b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.751+0000 7f7624695640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f761c0b53c0 0x7f761c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.751+0000 7f7624695640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f761c067e30 con 0x7f761c15ec10 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.751+0000 7f7624695640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f761c0b3c10 con 0x7f761c0b53c0 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.751+0000 7f7624695640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f761c0b3ef0 con 0x7f761c15e830 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7621c30640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f761c15ec10 0x7f761c067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:34764/0 (socket says 172.21.15.70:34764) 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7621c30640 1 -- 172.21.15.70:0/3854685876 learned_addr learned my addr 172.21.15.70:0/3854685876 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2521216445 0 0) 0x7f761c067e30 con 0x7f761c15ec10 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7604003520 con 0x7f761c15ec10 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622c32640 1 --2- 172.21.15.70:0/3854685876 >> v2:172.21.15.96:3300/0 conn(0x7f761c0b53c0 0x7f761c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f761c0b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3786898145 0 0) 0x7f7604003520 con 0x7f761c15ec10 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 msgr2=0x7f761c0b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f761c0b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 >> v2:172.21.15.96:3300/0 conn(0x7f761c0b53c0 msgr2=0x7f761c0b5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 --2- 172.21.15.70:0/3854685876 >> v2:172.21.15.96:3300/0 conn(0x7f761c0b53c0 0x7f761c0b5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c0b41d0 con 0x7f761c15ec10 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f7610001670 con 0x7f761c15ec10 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f761c0b4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 >> v1:172.21.15.70:6789/0 conn(0x7f761c15ec10 legacy=0x7f761c067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76040039b0 0x7f7604003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7604004800 0x7f7604004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f76040051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0b3c10 con 0x7f76040039b0 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0b3ef0 con 0x7f7604004800 2024-04-15T15:17:00.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f760400ba40 con 0x7f761c15e830 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76040039b0 0x7f7604003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7621c30640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7604004800 0x7f7604004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622c32640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f76040051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 -- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 msgr2=0x7f76040051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f761c15e830 0x7f76040051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 -- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7604004800 msgr2=0x7f7604004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.752+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f7604004800 0x7f7604004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.753+0000 7f7622431640 1 -- 172.21.15.70:0/3854685876 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7604003520 con 0x7f76040039b0 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.753+0000 7f7622431640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76040039b0 0x7f7604003dd0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f760c0073e0 tx=0x7f7604005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da57c30a063072b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.753+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c039070 con 0x7f76040039b0 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.753+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f760c02cd30 con 0x7f76040039b0 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.753+0000 7f762142f640 1 -- 172.21.15.70:0/3854685876 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f760c009c80 con 0x7f76040039b0 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 -- 172.21.15.70:0/3854685876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76040039b0 msgr2=0x7f7604003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 --2- 172.21.15.70:0/3854685876 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f76040039b0 0x7f7604003dd0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f760c0073e0 tx=0x7f7604005830 comp rx=0 tx=0).stop 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7622c32640 1 -- 172.21.15.70:0/3854685876 reap_dead start 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 -- 172.21.15.70:0/3854685876 shutdown_connections 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 -- 172.21.15.70:0/3854685876 >> 172.21.15.70:0/3854685876 conn(0x7f761c073360 msgr2=0x7f761c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 -- 172.21.15.70:0/3854685876 shutdown_connections 2024-04-15T15:17:00.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.754+0000 7f7624695640 1 -- 172.21.15.70:0/3854685876 wait complete. 2024-04-15T15:17:00.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.755+0000 7f7624695640 1 Processor -- start 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.755+0000 7f7624695640 1 -- start start 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f76040039b0 0x7f761c1423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c15b680 0x7f761c15a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f761c15ff60 con 0x7f761c0b0950 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f761c0b4790 con 0x7f761c15b680 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7624695640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f761c1600e0 con 0x7f76040039b0 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:46876/0 (socket says 172.21.15.70:46876) 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 -- 172.21.15.70:0/586698146 learned_addr learned my addr 172.21.15.70:0/586698146 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622431640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f76040039b0 0x7f761c1423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7621c30640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c15b680 0x7f761c15a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 -- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f76040039b0 msgr2=0x7f761c1423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f76040039b0 0x7f761c1423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 -- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c15b680 msgr2=0x7f761c15a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c15b680 0x7f761c15a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 -- 172.21.15.70:0/586698146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f761c142900 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.756+0000 7f7622c32640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f761c070eb0 tx=0x7f761400bac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.757+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614014680 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.757+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f761401a070 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.757+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f761c15bb80 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.757+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7614014a60 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.757+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f761c1582a0 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.758+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f761c0689e0 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.758+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f7614012040 con 0x7f761c0b0950 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.759+0000 7f761affd640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 0x7f760806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:00.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.759+0000 7f7622431640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 0x7f760806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.759+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f761408bc40 con 0x7f761c0b0950 2024-04-15T15:17:00.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.759+0000 7f7622431640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 0x7f760806ee70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f760c0073e0 tx=0x7f760c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:00.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.763+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f761c142900 con 0x7f761c0b0950 2024-04-15T15:17:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.975+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f761c06ff10 con 0x7f761c0b0950 2024-04-15T15:17:00.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.976+0000 7f761affd640 1 -- 172.21.15.70:0/586698146 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+5172 (secure 0 0 0) 0x7f7614059e80 con 0x7f761c0b0950 2024-04-15T15:17:00.978 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:00.978 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":17,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:00.630258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:replay","state_seq":1,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:17:00.978 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 17 2024-04-15T15:17:00.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 msgr2=0x7f760806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 0x7f760806ee70 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f760c0073e0 tx=0x7f760c030000 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 msgr2=0x7f761c0b0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f761c070eb0 tx=0x7f761400bac0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 shutdown_connections 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f760806c9b0 0x7f760806ee70 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f761c0b0950 0x7f761c0b0de0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f761c15b680 0x7f761c15a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 --2- 172.21.15.70:0/586698146 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f76040039b0 0x7f761c1423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 >> 172.21.15.70:0/586698146 conn(0x7f761c073360 msgr2=0x7f761c0afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 shutdown_connections 2024-04-15T15:17:00.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:00.978+0000 7f7624695640 1 -- 172.21.15.70:0/586698146 wait complete. 2024-04-15T15:17:01.393 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:25.228602+0000', 'modified': '2024-04-15T15:17:00.630258+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5518': {'gid': 5518, 'name': 'd', 'rank': 0, 'incarnation': 17, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.96:6835/1131645259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 1131645259}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1131645259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:17:01.393 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5518:up:replay 2024-04-15T15:17:02.394 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:02.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.507+0000 7fa783932640 1 Processor -- start 2024-04-15T15:17:02.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 -- start start 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa77c15c8d0 0x7fa77c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa77c09b0e0 0x7fa77c0b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa77c0d0350 con 0x7fa77c09b650 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa77c0b5b00 con 0x7fa77c09b0e0 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.508+0000 7fa783932640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa77c0ce100 con 0x7fa77c15c8d0 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa77c09b0e0 0x7fa77c0b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa77c09b0e0 0x7fa77c0b5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45264/0 (socket says 172.21.15.70:45264) 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 -- 172.21.15.70:0/2051228062 learned_addr learned my addr 172.21.15.70:0/2051228062 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa7816ce640 1 --2- 172.21.15.70:0/2051228062 >> v2:172.21.15.96:3300/0 conn(0x7fa77c15c8d0 0x7fa77c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 -- 172.21.15.70:0/2051228062 >> v2:172.21.15.96:3300/0 conn(0x7fa77c15c8d0 msgr2=0x7fa77c0cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:02.511 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 --2- 172.21.15.70:0/2051228062 >> v2:172.21.15.96:3300/0 conn(0x7fa77c15c8d0 0x7fa77c0cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 -- 172.21.15.70:0/2051228062 >> v1:172.21.15.96:6789/0 conn(0x7fa77c09b650 legacy=0x7fa77c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.512 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.509+0000 7fa781ecf640 1 -- 172.21.15.70:0/2051228062 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c0b54b0 con 0x7fa77c09b0e0 2024-04-15T15:17:02.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa781ecf640 1 --2- 172.21.15.70:0/2051228062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0b5f90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa77c09b620 tx=0x7fa77402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e721b5510ff60f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa77bfff640 1 -- 172.21.15.70:0/2051228062 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77402ca80 con 0x7fa77c09b0e0 2024-04-15T15:17:02.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa77bfff640 1 -- 172.21.15.70:0/2051228062 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77402cc20 con 0x7fa77c09b0e0 2024-04-15T15:17:02.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa77bfff640 1 -- 172.21.15.70:0/2051228062 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7740343f0 con 0x7fa77c09b0e0 2024-04-15T15:17:02.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 -- 172.21.15.70:0/2051228062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c09b0e0 msgr2=0x7fa77c0b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 --2- 172.21.15.70:0/2051228062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0b5f90 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa77c09b620 tx=0x7fa77402bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 -- 172.21.15.70:0/2051228062 shutdown_connections 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 --2- 172.21.15.70:0/2051228062 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0b5f90 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 --2- 172.21.15.70:0/2051228062 >> v2:172.21.15.96:3300/0 conn(0x7fa77c15c8d0 0x7fa77c0cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.510+0000 7fa783932640 1 -- 172.21.15.70:0/2051228062 >> 172.21.15.70:0/2051228062 conn(0x7fa77c0b3fb0 msgr2=0x7fa77c0b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.511+0000 7fa783932640 1 -- 172.21.15.70:0/2051228062 shutdown_connections 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.511+0000 7fa783932640 1 -- 172.21.15.70:0/2051228062 wait complete. 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 Processor -- start 2024-04-15T15:17:02.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 -- start start 2024-04-15T15:17:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0cc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa77c09b650 0x7fa77c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c0b6650 con 0x7fa77c0cade0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa77c0b64d0 con 0x7fa77c09b0e0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.512+0000 7fa783932640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa77c0b67d0 con 0x7fa77c09b650 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45280/0 (socket says 172.21.15.70:45280) 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 -- 172.21.15.70:0/797457429 learned_addr learned my addr 172.21.15.70:0/797457429 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa7816ce640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa780ecd640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa77c09b650 0x7fa77c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 -- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa77c09b650 msgr2=0x7fa77c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa77c09b650 0x7fa77c1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 -- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 msgr2=0x7fa77c0cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 -- 172.21.15.70:0/797457429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa77c0cb2c0 con 0x7fa77c0cade0 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa7816ce640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:02.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa781ecf640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa774009f40 tx=0x7fa774004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774043020 con 0x7fa77c0cade0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77403e450 con 0x7fa77c0cade0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa77c0cb5a0 con 0x7fa77c0cade0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa774009510 con 0x7fa77c0cade0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.513+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa77c0cbb60 con 0x7fa77c0cade0 2024-04-15T15:17:02.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.515+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa77c068b10 con 0x7fa77c0cade0 2024-04-15T15:17:02.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.515+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa774046020 con 0x7fa77c0cade0 2024-04-15T15:17:02.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.515+0000 7fa779ffb640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 0x7fa76806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:02.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.516+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa7740ae540 con 0x7fa77c0cade0 2024-04-15T15:17:02.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.516+0000 7fa7816ce640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 0x7fa76806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:02.519 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.516+0000 7fa7816ce640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 0x7fa76806ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00b9f0 tx=0x7fa76c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:02.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.520+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa77c068b10 con 0x7fa77c0cade0 2024-04-15T15:17:02.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.734+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa77c0690f0 con 0x7fa77c0cade0 2024-04-15T15:17:02.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.734+0000 7fa779ffb640 1 -- 172.21.15.70:0/797457429 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) ==== 94+0+5172 (secure 0 0 0) 0x7fa77404d030 con 0x7fa77c0cade0 2024-04-15T15:17:02.736 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:02.736 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":19,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":19,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:01.636283+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-15T15:17:02.736 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 19 2024-04-15T15:17:02.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 msgr2=0x7fa76806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 0x7fa76806ee40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa76c00b9f0 tx=0x7fa76c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 msgr2=0x7fa77c154c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fa774009f40 tx=0x7fa774004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 shutdown_connections 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa76806c980 0x7fa76806ee40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa77c0cade0 0x7fa77c154c10 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa77c09b650 0x7fa77c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 --2- 172.21.15.70:0/797457429 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa77c09b0e0 0x7fa77c0cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 >> 172.21.15.70:0/797457429 conn(0x7fa77c0b3fb0 msgr2=0x7fa77c0cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.736+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 shutdown_connections 2024-04-15T15:17:02.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:02.737+0000 7fa783932640 1 -- 172.21.15.70:0/797457429 wait complete. 2024-04-15T15:17:03.153 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 19, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:25.228602+0000', 'modified': '2024-04-15T15:17:01.636283+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5518': {'gid': 5518, 'name': 'd', 'rank': 0, 'incarnation': 17, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.96:6835/1131645259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 1131645259}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1131645259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-15T15:17:03.153 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5518:up:rejoin 2024-04-15T15:17:04.154 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.267+0000 7f3b58ad3640 1 Processor -- start 2024-04-15T15:17:04.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.267+0000 7f3b58ad3640 1 -- start start 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b58ad3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b540ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b58ad3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b540b1f40 con 0x7f3b540ced30 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b58ad3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b540cda30 con 0x7f3b540cf800 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b58ad3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540cdd10 con 0x7f3b5415fda0 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b51d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3b540ced30 0x7f3b540b1480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60324/0 (socket says 172.21.15.70:60324) 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.268+0000 7f3b51d9b640 1 -- 172.21.15.70:0/335070451 learned_addr learned my addr 172.21.15.70:0/335070451 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5259c640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b540ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961946360 0 0) 0x7f3b540b1f40 con 0x7f3b540ced30 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3b3c003520 con 0x7f3b540ced30 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3203446945 0 0) 0x7f3b3c003520 con 0x7f3b540ced30 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 msgr2=0x7f3b540ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b540ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 >> v1:172.21.15.96:6789/0 conn(0x7f3b540cf800 legacy=0x7f3b540cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b540cdff0 con 0x7f3b540ced30 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f3b48001670 con 0x7f3b540ced30 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 >> v1:172.21.15.70:6789/0 conn(0x7f3b540ced30 legacy=0x7f3b540b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b3c0039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b3c003f60 0x7f3b3c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540cdd10 con 0x7f3b3c004d40 2024-04-15T15:17:04.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3c00baa0 con 0x7f3b3c003f60 2024-04-15T15:17:04.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3b3c00bca0 con 0x7f3b5415fda0 2024-04-15T15:17:04.275 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b5259c640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b3c0039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b51d9b640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b3c003f60 0x7f3b3c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 -- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 msgr2=0x7f3b3c0039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b5415fda0 0x7f3b3c0039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 -- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b3c003f60 msgr2=0x7f3b3c004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b3c003f60 0x7f3b3c004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b52d9d640 1 -- 172.21.15.70:0/335070451 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b3c003520 con 0x7f3b3c004d40 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.269+0000 7f3b51d9b640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b3c003f60 0x7f3b3c004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.270+0000 7f3b52d9d640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c001750 tx=0x7f3b3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89f7140bd0b006e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.270+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c001370 con 0x7f3b3c004d40 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.270+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b4c0052d0 con 0x7f3b3c004d40 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.270+0000 7f3b5159a640 1 -- 172.21.15.70:0/335070451 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c003830 con 0x7f3b3c004d40 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 -- 172.21.15.70:0/335070451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3c004d40 msgr2=0x7f3b3c0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/335070451 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b3c004d40 0x7f3b3c0051c0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c001750 tx=0x7f3b3c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:04.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b52d9d640 1 -- 172.21.15.70:0/335070451 reap_dead start 2024-04-15T15:17:04.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 -- 172.21.15.70:0/335070451 shutdown_connections 2024-04-15T15:17:04.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 -- 172.21.15.70:0/335070451 >> 172.21.15.70:0/335070451 conn(0x7f3b540b6870 msgr2=0x7f3b540b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 -- 172.21.15.70:0/335070451 shutdown_connections 2024-04-15T15:17:04.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.271+0000 7f3b58ad3640 1 -- 172.21.15.70:0/335070451 wait complete. 2024-04-15T15:17:04.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.272+0000 7f3b58ad3640 1 Processor -- start 2024-04-15T15:17:04.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.272+0000 7f3b58ad3640 1 -- start start 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b3c004d40 0x7f3b5415ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b54155ed0 0x7f3b5415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3b540b0790 con 0x7f3b5414d0e0 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3b54160180 con 0x7f3b54155ed0 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b58ad3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3b540b1b90 con 0x7f3b3c004d40 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45306/0 (socket says 172.21.15.70:45306) 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 -- 172.21.15.70:0/2808615457 learned_addr learned my addr 172.21.15.70:0/2808615457 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b5259c640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b3c004d40 0x7f3b5415ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b52d9d640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b54155ed0 0x7f3b5415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 -- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b3c004d40 msgr2=0x7f3b5415ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b3c004d40 0x7f3b5415ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 -- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b54155ed0 msgr2=0x7f3b5415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.280 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.273+0000 7f3b51d9b640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b54155ed0 0x7f3b5415e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b51d9b640 1 -- 172.21.15.70:0/2808615457 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b54158040 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b51d9b640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800e800 tx=0x7f3b4800ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4800c870 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b48018070 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b541582e0 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4800cc10 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.274+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b541588a0 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.275+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b540689e0 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.275+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3b4801d460 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.276+0000 7f3b42ffd640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 0x7f3b3406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.276+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f3b4808bae0 con 0x7f3b5414d0e0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.276+0000 7f3b5259c640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 0x7f3b3406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:04.281 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.276+0000 7f3b5259c640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 0x7f3b3406ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b5415ec40 tx=0x7f3b4400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:04.282 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.280+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3b48059760 con 0x7f3b5414d0e0 2024-04-15T15:17:04.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.491+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b54157cf0 con 0x7f3b5414d0e0 2024-04-15T15:17:04.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.492+0000 7f3b42ffd640 1 -- 172.21.15.70:0/2808615457 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5179 (secure 0 0 0) 0x7f3b4805da60 con 0x7f3b5414d0e0 2024-04-15T15:17:04.494 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:04.494 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:03.637138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5518,"qdb_cluster":[5518]},"id":2}]} 2024-04-15T15:17:04.494 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 20 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 msgr2=0x7f3b3406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 0x7f3b3406ee40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3b5415ec40 tx=0x7f3b4400b040 comp rx=0 tx=0).stop 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 msgr2=0x7f3b5415e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3b4800e800 tx=0x7f3b4800ecc0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 shutdown_connections 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3b54155ed0 0x7f3b5415e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3b3406c980 0x7f3b3406ee40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3b5414d0e0 0x7f3b5415e060 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 --2- 172.21.15.70:0/2808615457 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3b3c004d40 0x7f3b5415ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:04.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 >> 172.21.15.70:0/2808615457 conn(0x7f3b540b6870 msgr2=0x7f3b5406f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:04.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 shutdown_connections 2024-04-15T15:17:04.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:04.494+0000 7f3b58ad3640 1 -- 172.21.15.70:0/2808615457 wait complete. 2024-04-15T15:17:04.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-15T15:16:25.228602+0000', 'modified': '2024-04-15T15:17:03.637138+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 42, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5518}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5518': {'gid': 5518, 'name': 'd', 'rank': 0, 'incarnation': 17, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.96:6835/1131645259', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6833', 'nonce': 1131645259}, {'type': 'v1', 'addr': '172.21.15.96:6835', 'nonce': 1131645259}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5518, 'qdb_cluster': [5518]} 2024-04-15T15:17:04.910 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-15T15:17:04.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-04-15T15:17:05.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.015+0000 7fe5da0b1640 1 Processor -- start 2024-04-15T15:17:05.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.015+0000 7fe5da0b1640 1 -- start start 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5da0b1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5d4071d10 0x7fe5d40d0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5da0b1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5d415db40 con 0x7fe5d4072d80 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5da0b1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5d4072170 con 0x7fe5d40d0bd0 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5da0b1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe5d40d1040 con 0x7fe5d4071d10 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d2ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe5d4071d10 0x7fe5d40d0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d37fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe5d4072d80 0x7fe5d40af3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60332/0 (socket says 172.21.15.70:60332) 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d37fe640 1 -- 172.21.15.70:0/1842904276 learned_addr learned my addr 172.21.15.70:0/1842904276 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:05.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1842904276 >> v1:172.21.15.96:6789/0 conn(0x7fe5d40d0bd0 legacy=0x7fe5d406e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1842904276 >> v1:172.21.15.70:6789/0 conn(0x7fe5d4072d80 legacy=0x7fe5d40af3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.016+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1842904276 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d415cfc0 con 0x7fe5d4071d10 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.017+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1842904276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4071d10 0x7fe5d40d0690 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80097a0 tx=0x7fe5c802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6616e493b9973e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.017+0000 7fe5d27fc640 1 -- 172.21.15.70:0/1842904276 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80315a0 con 0x7fe5d4071d10 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.017+0000 7fe5d27fc640 1 -- 172.21.15.70:0/1842904276 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8037070 con 0x7fe5d4071d10 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.017+0000 7fe5d27fc640 1 -- 172.21.15.70:0/1842904276 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8031960 con 0x7fe5d4071d10 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1842904276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4071d10 msgr2=0x7fe5d40d0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1842904276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4071d10 0x7fe5d40d0690 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80097a0 tx=0x7fe5c802b840 comp rx=0 tx=0).stop 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1842904276 shutdown_connections 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1842904276 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4071d10 0x7fe5d40d0690 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1842904276 >> 172.21.15.70:0/1842904276 conn(0x7fe5d406be00 msgr2=0x7fe5d406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.020 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1842904276 shutdown_connections 2024-04-15T15:17:05.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.018+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1842904276 wait complete. 2024-04-15T15:17:05.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.019+0000 7fe5da0b1640 1 Processor -- start 2024-04-15T15:17:05.021 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.019+0000 7fe5da0b1640 1 -- start start 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5d4071d10 0x7fe5d40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 0x7fe5d40cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d415d860 con 0x7fe5d4072d80 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d415d580 con 0x7fe5d40d0bd0 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5da0b1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe5d40afaf0 con 0x7fe5d4071d10 2024-04-15T15:17:05.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45312/0 (socket says 172.21.15.70:45312) 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1032622897 learned_addr learned my addr 172.21.15.70:0/1032622897 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5d4071d10 0x7fe5d40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d3fff640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 0x7fe5d40cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5d4071d10 msgr2=0x7fe5d40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5d4071d10 0x7fe5d40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 msgr2=0x7fe5d40cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 0x7fe5d40cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1032622897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5d40cd390 con 0x7fe5d4072d80 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d3fff640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 0x7fe5d40cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.020+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8034040 tx=0x7fe5c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.021+0000 7fe5bbfff640 1 -- 172.21.15.70:0/1032622897 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8032840 con 0x7fe5d4072d80 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.021+0000 7fe5bbfff640 1 -- 172.21.15.70:0/1032622897 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8032a00 con 0x7fe5d4072d80 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.021+0000 7fe5bbfff640 1 -- 172.21.15.70:0/1032622897 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8032df0 con 0x7fe5d4072d80 2024-04-15T15:17:05.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.021+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5d40cfc90 con 0x7fe5d4072d80 2024-04-15T15:17:05.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.021+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5d40c9ca0 con 0x7fe5d4072d80 2024-04-15T15:17:05.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.022+0000 7fe5bbfff640 1 -- 172.21.15.70:0/1032622897 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe5c803b3f0 con 0x7fe5d4072d80 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.023+0000 7fe5bbfff640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 0x7fe5c006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.023+0000 7fe5bbfff640 1 -- 172.21.15.70:0/1032622897 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe5c802f030 con 0x7fe5d4072d80 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.023+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 0x7fe5c006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.024+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 0x7fe5c006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40012e0 tx=0x7fe5c4001a30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.024+0000 7fe5dcd74740 1 Processor -- start 2024-04-15T15:17:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.024+0000 7fe5dcd74740 1 -- start start 2024-04-15T15:17:05.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc2e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc2e22f0 0x5646cc3120e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646cc24c9c0 con 0x5646cc30f1f0 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5646cc1f3650 con 0x5646cc2e22f0 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x5646cc248980 con 0x5646cc30ee10 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45314/0 (socket says 172.21.15.70:45314) 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1174033926 learned_addr learned my addr 172.21.15.70:0/1174033926 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 msgr2=0x5646cc2e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc2e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc2e22f0 msgr2=0x5646cc3120e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc2e22f0 0x5646cc3120e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/1174033926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646cbf5f2d0 con 0x5646cc30f1f0 2024-04-15T15:17:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c8032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75b922fdc499b25 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5b9ffb640 1 -- 172.21.15.70:0/1174033926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c80aa070 con 0x5646cc30f1f0 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5b9ffb640 1 -- 172.21.15.70:0/1174033926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8037050 con 0x5646cc30f1f0 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.025+0000 7fe5b9ffb640 1 -- 172.21.15.70:0/1174033926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c807b7e0 con 0x5646cc30f1f0 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 -- 172.21.15.70:0/1174033926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 msgr2=0x5646cc311ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c8032f90 comp rx=0 tx=0).stop 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 -- 172.21.15.70:0/1174033926 shutdown_connections 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc2e22f0 0x5646cc3120e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc311ba0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/1174033926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc2e1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 -- 172.21.15.70:0/1174033926 >> 172.21.15.70:0/1174033926 conn(0x5646cc2e4020 msgr2=0x5646cc32cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 -- 172.21.15.70:0/1174033926 shutdown_connections 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.026+0000 7fe5dcd74740 1 -- 172.21.15.70:0/1174033926 wait complete. 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 -- start start 2024-04-15T15:17:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 Processor -- start 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc310d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc323ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646cc24c9c0 con 0x5646cc30f1f0 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5646cc206a10 con 0x5646cc30ee10 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x5646cc2e65c0 con 0x5646cc313d30 2024-04-15T15:17:05.034 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5dcd74740 10 client.0 _ll_register_callbacks cb 0x5646cc242250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.027+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45320/0 (socket says 172.21.15.70:45320) 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/2459090382 learned_addr learned my addr 172.21.15.70:0/2459090382 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d3fff640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc323ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc310d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 msgr2=0x5646cc323ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc323ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 msgr2=0x5646cc310d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc310d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/2459090382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646cc0296b0 con 0x5646cc30f1f0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80ac070 tx=0x7fe5c8046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1fc81f2897f5af2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8098070 con 0x5646cc30f1f0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe5c8084140 con 0x5646cc30f1f0 2024-04-15T15:17:05.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5c8037050 con 0x5646cc30f1f0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe5c80a4050 con 0x5646cc30f1f0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5dcd74740 10 client.6075 fetch_fsmap learned FSMap version 20 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) -- 0x5646cc0296b0 con 0x5646cc30f1f0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8037050 con 0x5646cc30f1f0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 20) ==== 3494+0+0 (secure 0 0 0) 0x7fe5c8093070 con 0x5646cc30f1f0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5dcd74740 10 client.6075 fetch_fsmap finished waiting for FSMap version 20 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.028+0000 7fe5dcd74740 10 client.6075 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5dcd74740 20 client.6075 populate_metadata read hostname 'smithi070' 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc310250 0x5646cc340310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5dcd74740 4 client.6075 mds_command: new command op to 5518 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 --> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5646cc25a3e0 con 0x5646cc310250 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc310250 0x5646cc340310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5b27fc640 10 client.6075 ms_handle_connect on v2:172.21.15.96:6833/1131645259 2024-04-15T15:17:05.036 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.029+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc310250 0x5646cc340310 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d18c5344612f86bc server_cookie=3f6aef470d5b8585 in_seq=0 out_seq=0 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5b27fc640 1 -- 172.21.15.70:0/2459090382 <== mds.0 v2:172.21.15.96:6833/1131645259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5646cc25a3e0 con 0x5646cc310250 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5b27fc640 10 client.6075 handle_command_reply: tid=0 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 client.6075 shutdown 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 msgr2=0x5646cc315500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80ac070 tx=0x7fe5c8046000 comp rx=0 tx=0).stop 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 shutdown_connections 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc323ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc315500 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc310250 0x5646cc340310 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/2459090382 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc310d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 >> 172.21.15.70:0/2459090382 conn(0x5646cc2e22f0 msgr2=0x5646cc3300b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5b27fc640 0 client.6075 ms_handle_reset on v2:172.21.15.96:6833/1131645259 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 shutdown_connections 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 1 -- 172.21.15.70:0/2459090382 wait complete. 2024-04-15T15:17:05.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.030+0000 7fe5dcd74740 20 client.6075 trim_cache size 0 max 16384 2024-04-15T15:17:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.048+0000 7fe5dcd74740 1 Processor -- start 2024-04-15T15:17:05.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.048+0000 7fe5dcd74740 1 -- start start 2024-04-15T15:17:05.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc311770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc3131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d4200640 con 0x5646cc30f1f0 2024-04-15T15:17:05.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe5d41ff9a0 con 0x5646cc30ee10 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe5d41ffff0 con 0x5646cc313d30 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45322/0 (socket says 172.21.15.70:45322) 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3333441896 learned_addr learned my addr 172.21.15.70:0/3333441896 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 msgr2=0x5646cc3131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc3131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 msgr2=0x5646cc311770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc311770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3333441896 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5646cc312630 con 0x5646cc30f1f0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.049+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c807d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57929ec857a3dfe2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5b97fa640 1 -- 172.21.15.70:0/3333441896 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8088070 con 0x5646cc30f1f0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5b97fa640 1 -- 172.21.15.70:0/3333441896 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8037040 con 0x5646cc30f1f0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5b97fa640 1 -- 172.21.15.70:0/3333441896 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8098070 con 0x5646cc30f1f0 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3333441896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 msgr2=0x5646cc312c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.053 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c807d000 comp rx=0 tx=0).stop 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3333441896 shutdown_connections 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc313d30 0x5646cc3131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc312c70 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3333441896 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc30ee10 0x5646cc311770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3333441896 >> 172.21.15.70:0/3333441896 conn(0x5646cc2e22f0 msgr2=0x5646cc33f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3333441896 shutdown_connections 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.050+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3333441896 wait complete. 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 -- start start 2024-04-15T15:17:05.054 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 Processor -- start 2024-04-15T15:17:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc39d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc313d30 0x5646cc312c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5646cc24c9c0 con 0x5646cc30f1f0 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x5646cc2e65c0 con 0x5646cc313d30 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.051+0000 7fe5dcd74740 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x5646cc206a10 con 0x5646cc30ee10 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5dcd74740 10 client.0 _ll_register_callbacks cb 0x7fe5c80a8070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45330/0 (socket says 172.21.15.70:45330) 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3245755345 learned_addr learned my addr 172.21.15.70:0/3245755345 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc39d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d3fff640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc313d30 0x5646cc312c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 msgr2=0x5646cc39d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc39d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc313d30 msgr2=0x5646cc312c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc313d30 0x5646cc312c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 -- 172.21.15.70:0/3245755345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5646cc346db0 con 0x5646cc30f1f0 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5d2ffd640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b04278caf9eb96 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.70:3300/0 2024-04-15T15:17:05.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5c8088040 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mon.0 v2:172.21.15.70:3300/0 2 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe5c807ae40 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe5c8037040 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe5c80aa030 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5dcd74740 10 client.6081 fetch_fsmap learned FSMap version 20 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) -- 0x5646cc346db0 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mon.0 v2:172.21.15.70:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5c8037040 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mon.0 v2:172.21.15.70:3300/0 5 ==== fsmap(e 20) ==== 3494+0+0 (secure 0 0 0) 0x7fe5c8093040 con 0x5646cc30f1f0 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.052+0000 7fe5dcd74740 10 client.6081 fetch_fsmap finished waiting for FSMap version 20 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 10 client.6081 resolve_mds: resolved name 'd' to daemon mds.d 2024-04-15T15:17:05.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 20 client.6081 populate_metadata read hostname 'smithi070' 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc3300b0 0x5646cc330490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 4 client.6081 mds_command: new command op to 5518 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 --> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] -- command(tid 0: {"prefix": "status"}) -- 0x5646cc254250 con 0x5646cc3300b0 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc3300b0 0x5646cc330490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5b27fc640 10 client.6081 ms_handle_connect on v2:172.21.15.96:6833/1131645259 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5d37fe640 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc3300b0 0x5646cc330490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fc86b57bedf7ee1 server_cookie=7283e702995a55e in_seq=0 out_seq=0 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5b27fc640 1 -- 172.21.15.70:0/3245755345 <== mds.0 v2:172.21.15.96:6833/1131645259 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x5646cc254250 con 0x5646cc3300b0 2024-04-15T15:17:05.062 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5b27fc640 10 client.6081 handle_command_reply: tid=0 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 client.6081 shutdown 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 msgr2=0x5646cc2e7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe5c807b4e0 tx=0x7fe5c8009190 comp rx=0 tx=0).stop 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 shutdown_connections 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:6833/1131645259,v1:172.21.15.96:6835/1131645259] conn(0x5646cc3300b0 0x5646cc330490 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x5646cc313d30 0x5646cc312c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x5646cc30f1f0 0x5646cc2e7230 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5b27fc640 0 client.6081 ms_handle_reset on v2:172.21.15.96:6833/1131645259 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 --2- 172.21.15.70:0/3245755345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x5646cc30ee10 0x5646cc39d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.063 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.053+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 >> 172.21.15.70:0/3245755345 conn(0x5646cc2e22f0 msgr2=0x5646cc3238d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 shutdown_connections 2024-04-15T15:17:05.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5dcd74740 1 -- 172.21.15.70:0/3245755345 wait complete. 2024-04-15T15:17:05.064 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5dcd74740 20 client.6081 trim_cache size 0 max 16384 2024-04-15T15:17:05.064 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "cluster_fsid": "074678c8-a3ad-4fb8-89f1-2c7d457fbbe5", 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "whoami": 0, 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "id": 5518, 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "want_state": "up:active", 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "state": "up:active", 2024-04-15T15:17:05.065 INFO:teuthology.orchestra.run.smithi070.stdout: "fs_name": "cephfs", 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout: "rank_uptime": 4.4225630300000001, 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout: "mdsmap_epoch": 20, 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch": 42, 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout: "osdmap_epoch_barrier": 42, 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout: "uptime": 4.9271835189999997 2024-04-15T15:17:05.066 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:05.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 msgr2=0x7fe5c006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 0x7fe5c006ee40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe5c40012e0 tx=0x7fe5c4001a30 comp rx=0 tx=0).stop 2024-04-15T15:17:05.069 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 msgr2=0x7fe5d40b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8034040 tx=0x7fe5c80046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 shutdown_connections 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe5d40d0bd0 0x7fe5d40cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe5c006c980 0x7fe5c006ee40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5d4072d80 0x7fe5d40b1e40 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.070 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 --2- 172.21.15.70:0/1032622897 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe5d4071d10 0x7fe5d40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 >> 172.21.15.70:0/1032622897 conn(0x7fe5d406be00 msgr2=0x7fe5d4159260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 shutdown_connections 2024-04-15T15:17:05.071 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.054+0000 7fe5da0b1640 1 -- 172.21.15.70:0/1032622897 wait complete. 2024-04-15T15:17:05.072 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-15T15:17:05.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:17:05.111 DEBUG:teuthology.orchestra.run.smithi096:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-15T15:17:05.173 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-15T15:17:05.173 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:05.173 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:17:05.173 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:05.190 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:05.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.330+0000 7f3842c77640 1 Processor -- start 2024-04-15T15:17:05.332 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.330+0000 7f3842c77640 1 -- start start 2024-04-15T15:17:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f383c072d80 0x7f383c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c0af3e0 0x7f383c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0d0690 0x7f383c0d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f383c15db40 con 0x7f383c0af3e0 2024-04-15T15:17:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f383c15cfc0 con 0x7f383c072d80 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f3842c77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f383c15d2a0 con 0x7f383c0d0690 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f383bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c0af3e0 0x7f383c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f383bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f383c0af3e0 0x7f383c0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45334/0 (socket says 172.21.15.70:45334) 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.331+0000 7f383bfff640 1 -- 172.21.15.70:0/2348892718 learned_addr learned my addr 172.21.15.70:0/2348892718 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f3841214640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0d0690 0x7f383c0d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f3840a13640 1 --2- 172.21.15.70:0/2348892718 >> v2:172.21.15.96:3300/0 conn(0x7f383c072d80 0x7f383c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 -- 172.21.15.70:0/2348892718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0d0690 msgr2=0x7f383c0d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0d0690 0x7f383c0d0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 -- 172.21.15.70:0/2348892718 >> v2:172.21.15.96:3300/0 conn(0x7f383c072d80 msgr2=0x7f383c071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 --2- 172.21.15.70:0/2348892718 >> v2:172.21.15.96:3300/0 conn(0x7f383c072d80 0x7f383c071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 -- 172.21.15.70:0/2348892718 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c15d580 con 0x7f383c0af3e0 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.332+0000 7f383bfff640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0af3e0 0x7f383c0af840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f383c0d11c0 tx=0x7f382402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69603eeb36062543 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f383b7fe640 1 -- 172.21.15.70:0/2348892718 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382402c9e0 con 0x7f383c0af3e0 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f383b7fe640 1 -- 172.21.15.70:0/2348892718 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f382402cb80 con 0x7f383c0af3e0 2024-04-15T15:17:05.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f383b7fe640 1 -- 172.21.15.70:0/2348892718 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38240343f0 con 0x7f383c0af3e0 2024-04-15T15:17:05.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 -- 172.21.15.70:0/2348892718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0af3e0 msgr2=0x7f383c0af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0af3e0 0x7f383c0af840 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f383c0d11c0 tx=0x7f382402be40 comp rx=0 tx=0).stop 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 -- 172.21.15.70:0/2348892718 shutdown_connections 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c0d0690 0x7f383c0d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 --2- 172.21.15.70:0/2348892718 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0af3e0 0x7f383c0af840 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 --2- 172.21.15.70:0/2348892718 >> v2:172.21.15.96:3300/0 conn(0x7f383c072d80 0x7f383c071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 -- 172.21.15.70:0/2348892718 >> 172.21.15.70:0/2348892718 conn(0x7f383c06be00 msgr2=0x7f383c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 -- 172.21.15.70:0/2348892718 shutdown_connections 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.333+0000 7f3842c77640 1 -- 172.21.15.70:0/2348892718 wait complete. 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.334+0000 7f3842c77640 1 Processor -- start 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.334+0000 7f3842c77640 1 -- start start 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c072d80 0x7f383c159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 0x7f383c153a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.340 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15cd10 con 0x7f383c0d0690 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f383c15cb90 con 0x7f383c0af3e0 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3842c77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f383c06e060 con 0x7f383c072d80 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3841214640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3841214640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45338/0 (socket says 172.21.15.70:45338) 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.335+0000 7f3841214640 1 -- 172.21.15.70:0/875062687 learned_addr learned my addr 172.21.15.70:0/875062687 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f383bfff640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 0x7f383c153a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3840a13640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c072d80 0x7f383c159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.341 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 -- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c072d80 msgr2=0x7f383c159e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c072d80 0x7f383c159e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 -- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 msgr2=0x7f383c153a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 0x7f383c153a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 -- 172.21.15.70:0/875062687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c153f50 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f383bfff640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 0x7f383c153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3841214640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f383400d600 tx=0x7f383400dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38340125f0 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3834019070 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38340129f0 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c1529b0 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.336+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c152f30 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.337+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c0689e0 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.337+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3834012b90 con 0x7f383c0d0690 2024-04-15T15:17:05.342 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.338+0000 7f38397fa640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 0x7f381c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.338+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f383408c180 con 0x7f383c0d0690 2024-04-15T15:17:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.338+0000 7f3840a13640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 0x7f381c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.338+0000 7f3840a13640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 0x7f381c06ee40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f382c006fd0 tx=0x7f382c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.342+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f383c153f50 con 0x7f383c0d0690 2024-04-15T15:17:05.522 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.520+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f383c0ad600 con 0x7f383c0d0690 2024-04-15T15:17:05.523 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.521+0000 7f38397fa640 1 -- 172.21.15.70:0/875062687 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7f383405a3d0 con 0x7f383c0d0690 2024-04-15T15:17:05.523 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:05.524 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 msgr2=0x7f381c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 0x7f381c06ee40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f382c006fd0 tx=0x7f382c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 msgr2=0x7f383c1f2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f383400d600 tx=0x7f383400dac0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 shutdown_connections 2024-04-15T15:17:05.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f381c06c980 0x7f381c06ee40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f383c0d0690 0x7f383c1f2220 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f383c0af3e0 0x7f383c153a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 --2- 172.21.15.70:0/875062687 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f383c072d80 0x7f383c159e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.523+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 >> 172.21.15.70:0/875062687 conn(0x7f383c06be00 msgr2=0x7f383c071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.524+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 shutdown_connections 2024-04-15T15:17:05.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.524+0000 7f3842c77640 1 -- 172.21.15.70:0/875062687 wait complete. 2024-04-15T15:17:05.539 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:05.540 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:05.638 INFO:tasks.ceph.mds.d.smithi096.stderr:2024-04-15T15:17:05.636+0000 7f9636f21640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:05.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.646+0000 7fd202f9d640 1 Processor -- start 2024-04-15T15:17:05.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.646+0000 7fd202f9d640 1 -- start start 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd202f9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1fc09a9d0 0x7fd1fc160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd202f9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 0x7fd1fc0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd202f9d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1fc070720 con 0x7fd1fc0701f0 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd202f9d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd1fc09ba90 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd202f9d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd1fc0d08c0 con 0x7fd1fc15fb30 2024-04-15T15:17:05.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd200d39640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd1fc09a9d0 0x7fd1fc160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd1fbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd1fc0701f0 0x7fd1fc09adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60348/0 (socket says 172.21.15.70:60348) 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd1fbfff640 1 -- 172.21.15.70:0/2560442785 learned_addr learned my addr 172.21.15.70:0/2560442785 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd20153a640 1 --2- 172.21.15.70:0/2560442785 >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 0x7fd1fc0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd200d39640 1 -- 172.21.15.70:0/2560442785 >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 msgr2=0x7fd1fc0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd200d39640 1 --2- 172.21.15.70:0/2560442785 >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 0x7fd1fc0d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd1fb7fe640 1 -- 172.21.15.70:0/2560442785 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4184329076 0 0) 0x7fd1fc070720 con 0x7fd1fc0701f0 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd200d39640 1 -- 172.21.15.70:0/2560442785 >> v1:172.21.15.70:6789/0 conn(0x7fd1fc0701f0 legacy=0x7fd1fc09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd200d39640 1 -- 172.21.15.70:0/2560442785 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc070a90 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.647+0000 7fd20153a640 1 --2- 172.21.15.70:0/2560442785 >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 0x7fd1fc0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:05.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd200d39640 1 --2- 172.21.15.70:0/2560442785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc160420 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc073740 tx=0x7fd1f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf605b0d00ab9aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd1fb7fe640 1 -- 172.21.15.70:0/2560442785 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0035020 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd1fb7fe640 1 -- 172.21.15.70:0/2560442785 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1f0031830 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd1fb7fe640 1 -- 172.21.15.70:0/2560442785 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0031c00 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd202f9d640 1 -- 172.21.15.70:0/2560442785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 msgr2=0x7fd1fc160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.648+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2560442785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc160420 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fd1fc073740 tx=0x7fd1f002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 -- 172.21.15.70:0/2560442785 shutdown_connections 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2560442785 >> v2:172.21.15.96:3300/0 conn(0x7fd1fc15fb30 0x7fd1fc0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2560442785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc160420 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 -- 172.21.15.70:0/2560442785 >> 172.21.15.70:0/2560442785 conn(0x7fd1fc0d2460 msgr2=0x7fd1fc0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 -- 172.21.15.70:0/2560442785 shutdown_connections 2024-04-15T15:17:05.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.649+0000 7fd202f9d640 1 -- 172.21.15.70:0/2560442785 wait complete. 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.650+0000 7fd202f9d640 1 Processor -- start 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.650+0000 7fd202f9d640 1 -- start start 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd1fc0701f0 0x7fd1fc0b7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1fc15fb30 0x7fd1fc0b0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc15ffc0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd1fc071050 con 0x7fd1fc15fb30 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd202f9d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd1fc0b1c60 con 0x7fd1fc0701f0 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45366/0 (socket says 172.21.15.70:45366) 2024-04-15T15:17:05.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 -- 172.21.15.70:0/2170381298 learned_addr learned my addr 172.21.15.70:0/2170381298 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd200d39640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd1fc0701f0 0x7fd1fc0b7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd20153a640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1fc15fb30 0x7fd1fc0b0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 -- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd1fc0701f0 msgr2=0x7fd1fc0b7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd1fc0701f0 0x7fd1fc0b7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 -- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1fc15fb30 msgr2=0x7fd1fc0b0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1fc15fb30 0x7fd1fc0b0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 -- 172.21.15.70:0/2170381298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1fc0afb50 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.651+0000 7fd1fbfff640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800c440 tx=0x7fd1e800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.652+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8018070 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.652+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1e80074e0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.652+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8004a20 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.652+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1fc0afe30 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.652+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1fc0b03f0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.653+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1fc0689e0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.653+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd1e801b070 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.654+0000 7fd1f97fa640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 0x7fd1e006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.654+0000 7fd200d39640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 0x7fd1e006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.654+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd1e808afe0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.661 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.654+0000 7fd200d39640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 0x7fd1e006ee40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0036000 tx=0x7fd1f003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.662 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.658+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd1e8058bd0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.834+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd1fc0c92a0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.834+0000 7fd1f97fa640 1 -- 172.21.15.70:0/2170381298 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fd1e805ced0 con 0x7fd1fc09a9d0 2024-04-15T15:17:05.837 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:17:05.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 msgr2=0x7fd1e006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 0x7fd1e006ee40 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd1f0036000 tx=0x7fd1f003d040 comp rx=0 tx=0).stop 2024-04-15T15:17:05.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 msgr2=0x7fd1fc0af300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd1e800c440 tx=0x7fd1e800c900 comp rx=0 tx=0).stop 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 shutdown_connections 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd1fc15fb30 0x7fd1fc0b0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd1e006c980 0x7fd1e006ee40 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd1fc09a9d0 0x7fd1fc0af300 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 --2- 172.21.15.70:0/2170381298 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd1fc0701f0 0x7fd1fc0b7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.836+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 >> 172.21.15.70:0/2170381298 conn(0x7fd1fc0d2460 msgr2=0x7fd1fc06e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.837+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 shutdown_connections 2024-04-15T15:17:05.840 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.837+0000 7fd202f9d640 1 -- 172.21.15.70:0/2170381298 wait complete. 2024-04-15T15:17:05.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:05.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.956+0000 7fad9ddf2640 1 Processor -- start 2024-04-15T15:17:05.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.956+0000 7fad9ddf2640 1 -- start start 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad9ddf2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fad980af290 0x7fad98166300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad9ddf2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad9815cb50 con 0x7fad980b3160 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad9ddf2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad9815bfd0 con 0x7fad980d0690 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad9ddf2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fad9815bda0 con 0x7fad980af290 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad96ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fad980b3160 0x7fad9815b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60362/0 (socket says 172.21.15.70:60362) 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad96ffd640 1 -- 172.21.15.70:0/1581576860 learned_addr learned my addr 172.21.15.70:0/1581576860 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad97fff640 1 --2- 172.21.15.70:0/1581576860 >> v2:172.21.15.96:3300/0 conn(0x7fad980af290 0x7fad98166300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274384240 0 0) 0x7fad9815cb50 con 0x7fad980b3160 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad80003520 con 0x7fad980b3160 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 876964466 0 0) 0x7fad9815bfd0 con 0x7fad980d0690 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fad9815cb50 con 0x7fad980d0690 2024-04-15T15:17:05.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1111463849 0 0) 0x7fad80003520 con 0x7fad980b3160 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 >> v2:172.21.15.96:3300/0 conn(0x7fad980af290 msgr2=0x7fad98166300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 --2- 172.21.15.70:0/1581576860 >> v2:172.21.15.96:3300/0 conn(0x7fad980af290 0x7fad98166300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 >> v1:172.21.15.96:6789/0 conn(0x7fad980d0690 legacy=0x7fad980d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9815c2b0 con 0x7fad980b3160 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fad8c001670 con 0x7fad980b3160 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 >> v1:172.21.15.70:6789/0 conn(0x7fad980b3160 legacy=0x7fad9815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad80003b70 0x7fad80003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad800051c0 0x7fad8000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad8000e240 con 0x7fad80004800 2024-04-15T15:17:05.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad8000e3f0 con 0x7fad80003b70 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fad8000e5f0 con 0x7fad800051c0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad96ffd640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad97fff640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad800051c0 0x7fad8000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.957+0000 7fad977fe640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad80003b70 0x7fad80003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 -- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad800051c0 msgr2=0x7fad8000dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad800051c0 0x7fad8000dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 -- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad80003b70 msgr2=0x7fad80003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad80003b70 0x7fad80003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 -- 172.21.15.70:0/1581576860 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad80003520 con 0x7fad80004800 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad977fe640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad80003b70 0x7fad80003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad97fff640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad800051c0 0x7fad8000dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad97fff640 1 -- 172.21.15.70:0/1581576860 reap_dead start 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad96ffd640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 0x7fad80004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c190 tx=0x7fad80005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3e611b29b922a68 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c007cc0 con 0x7fad80004800 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.958+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c030070 con 0x7fad80004800 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad967fc640 1 -- 172.21.15.70:0/1581576860 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c005d70 con 0x7fad80004800 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 -- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 msgr2=0x7fad80004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 0x7fad80004c80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c190 tx=0x7fad80005830 comp rx=0 tx=0).stop 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 -- 172.21.15.70:0/1581576860 shutdown_connections 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/1581576860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad80004800 0x7fad80004c80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 -- 172.21.15.70:0/1581576860 >> 172.21.15.70:0/1581576860 conn(0x7fad9806be00 msgr2=0x7fad9806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 -- 172.21.15.70:0/1581576860 shutdown_connections 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.959+0000 7fad9ddf2640 1 -- 172.21.15.70:0/1581576860 wait complete. 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.960+0000 7fad9ddf2640 1 Processor -- start 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 -- start start 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad80004800 0x7fad9815d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad9815f2d0 0x7fad9815f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad980b4fb0 con 0x7fad9806e0d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fad9815c870 con 0x7fad9815f2d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad9ddf2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fad9815b690 con 0x7fad80004800 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad96ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45382/0 (socket says 172.21.15.70:45382) 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.961+0000 7fad96ffd640 1 -- 172.21.15.70:0/2883806460 learned_addr learned my addr 172.21.15.70:0/2883806460 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 -- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad80004800 msgr2=0x7fad9815d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad80004800 0x7fad9815d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 -- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad9815f2d0 msgr2=0x7fad9815f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad9815f2d0 0x7fad9815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 -- 172.21.15.70:0/2883806460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad981452d0 con 0x7fad9806e0d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad96ffd640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c1f0 tx=0x7fad8c0019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c001c00 con 0x7fad9806e0d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad8c007480 con 0x7fad9806e0d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad981455b0 con 0x7fad9806e0d0 2024-04-15T15:17:05.973 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c02c770 con 0x7fad9806e0d0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.962+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad98145b70 con 0x7fad9806e0d0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.963+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad980689e0 con 0x7fad9806e0d0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.963+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fad8c048020 con 0x7fad9806e0d0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.964+0000 7fad7ffff640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 0x7fad8406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.964+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fad8c0ad1e0 con 0x7fad9806e0d0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.964+0000 7fad977fe640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 0x7fad8406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:05.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.964+0000 7fad977fe640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 0x7fad8406ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad88000f60 tx=0x7fad88009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:05.975 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:05.968+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fad8c07b4b0 con 0x7fad9806e0d0 2024-04-15T15:17:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.144+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fad980cc930 con 0x7fad9806e0d0 2024-04-15T15:17:06.146 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.144+0000 7fad7ffff640 1 -- 172.21.15.70:0/2883806460 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fad8c07adb0 con 0x7fad9806e0d0 2024-04-15T15:17:06.147 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:17:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 msgr2=0x7fad8406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 0x7fad8406ee70 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fad88000f60 tx=0x7fad88009040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 msgr2=0x7fad9806e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00c1f0 tx=0x7fad8c0019c0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 shutdown_connections 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fad9815f2d0 0x7fad9815f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fad8406c9b0 0x7fad8406ee70 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9806e0d0 0x7fad9806e550 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 --2- 172.21.15.70:0/2883806460 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fad80004800 0x7fad9815d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.146+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 >> 172.21.15.70:0/2883806460 conn(0x7fad9806be00 msgr2=0x7fad98074dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.147+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 shutdown_connections 2024-04-15T15:17:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.147+0000 7fad9ddf2640 1 -- 172.21.15.70:0/2883806460 wait complete. 2024-04-15T15:17:06.160 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:06.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.264+0000 7f948d2df640 1 Processor -- start 2024-04-15T15:17:06.266 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.264+0000 7f948d2df640 1 -- start start 2024-04-15T15:17:06.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f948d2df640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94880d2360 0x7f94880d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f948d2df640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94880b47a0 con 0x7f948815eae0 2024-04-15T15:17:06.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f948d2df640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94880b4a80 con 0x7f948815e700 2024-04-15T15:17:06.270 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f948d2df640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f94880b53b0 con 0x7f94880d2360 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94877fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f948815eae0 0x7f94880cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60366/0 (socket says 172.21.15.70:60366) 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94877fe640 1 -- 172.21.15.70:0/200629521 learned_addr learned my addr 172.21.15.70:0/200629521 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f947ffff640 1 --2- 172.21.15.70:0/200629521 >> v2:172.21.15.96:3300/0 conn(0x7f94880d2360 0x7f94880d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3894485989 0 0) 0x7f94880b47a0 con 0x7f948815eae0 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9474003520 con 0x7f948815eae0 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3978611661 0 0) 0x7f9474003520 con 0x7f948815eae0 2024-04-15T15:17:06.271 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 >> v2:172.21.15.96:3300/0 conn(0x7f94880d2360 msgr2=0x7f94880d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 --2- 172.21.15.70:0/200629521 >> v2:172.21.15.96:3300/0 conn(0x7f94880d2360 0x7f94880d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 >> v1:172.21.15.96:6789/0 conn(0x7f948815e700 legacy=0x7f94880b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94880b4d60 con 0x7f948815eae0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.265+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f94800019f0 con 0x7f948815eae0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 >> v1:172.21.15.70:6789/0 conn(0x7f948815eae0 legacy=0x7f94880cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f9474003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9474004800 0x7f9474004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94740051c0 0x7f947400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94880b53b0 con 0x7f94740039b0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f947400c090 con 0x7f9474004800 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f947400c290 con 0x7f94740051c0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f9474003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f947ffff640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9474004800 0x7f9474004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.272 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94877fe640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94740051c0 0x7f947400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 -- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94740051c0 msgr2=0x7f947400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94740051c0 0x7f947400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 -- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9474004800 msgr2=0x7f9474004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9474004800 0x7f9474004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f9486ffd640 1 -- 172.21.15.70:0/200629521 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9474003520 con 0x7f94740039b0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.266+0000 7f94877fe640 1 -- 172.21.15.70:0/200629521 reap_dead start 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f9486ffd640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f9474003df0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9478001770 tx=0x7f9474005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f29e63ad305ef3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94780013c0 con 0x7f94740039b0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94780052d0 con 0x7f94740039b0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f94867fc640 1 -- 172.21.15.70:0/200629521 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478003640 con 0x7f94740039b0 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 -- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 msgr2=0x7f9474003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f9474003df0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f9478001770 tx=0x7f9474005830 comp rx=0 tx=0).stop 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 -- 172.21.15.70:0/200629521 shutdown_connections 2024-04-15T15:17:06.273 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 --2- 172.21.15.70:0/200629521 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f9474003df0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 -- 172.21.15.70:0/200629521 >> 172.21.15.70:0/200629521 conn(0x7f948808bf60 msgr2=0x7f94880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.267+0000 7f948d2df640 1 -- 172.21.15.70:0/200629521 shutdown_connections 2024-04-15T15:17:06.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.268+0000 7f948d2df640 1 -- 172.21.15.70:0/200629521 wait complete. 2024-04-15T15:17:06.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 Processor -- start 2024-04-15T15:17:06.274 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 -- start start 2024-04-15T15:17:06.276 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94880c9780 0x7f94880b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94881f8020 0x7f948815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94880d1d40 con 0x7f94740039b0 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f94880b4450 con 0x7f94881f8020 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f948d2df640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f94880d1ec0 con 0x7f94880c9780 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f9486ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45392/0 (socket says 172.21.15.70:45392) 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.269+0000 7f9486ffd640 1 -- 172.21.15.70:0/1123048811 learned_addr learned my addr 172.21.15.70:0/1123048811 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f94877fe640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94881f8020 0x7f948815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f947ffff640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94880c9780 0x7f94880b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 -- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94880c9780 msgr2=0x7f94880b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94880c9780 0x7f94880b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 -- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94881f8020 msgr2=0x7f948815ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94881f8020 0x7f948815ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 -- 172.21.15.70:0/1123048811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94880b98a0 con 0x7f94740039b0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f9486ffd640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9478001890 tx=0x7f9478001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478041070 con 0x7f94740039b0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94780050d0 con 0x7f94740039b0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9478007470 con 0x7f94740039b0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94880b9b80 con 0x7f94740039b0 2024-04-15T15:17:06.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.270+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9488153b00 con 0x7f94740039b0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.271+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94880689e0 con 0x7f94740039b0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.272+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9478044070 con 0x7f94740039b0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.272+0000 7f947f7fe640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 0x7f946806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.272+0000 7f947ffff640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 0x7f946806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.272+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f94780ad940 con 0x7f94740039b0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.273+0000 7f947ffff640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 0x7f946806ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9488154170 tx=0x7f9470008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.279 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.277+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f94780052a0 con 0x7f94740039b0 2024-04-15T15:17:06.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.451+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f94880d5320 con 0x7f94740039b0 2024-04-15T15:17:06.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.452+0000 7f947f7fe640 1 -- 172.21.15.70:0/1123048811 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f947807ba70 con 0x7f94740039b0 2024-04-15T15:17:06.454 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:17:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 msgr2=0x7f946806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 0x7f946806ee40 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9488154170 tx=0x7f9470008040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 msgr2=0x7f94881535c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9478001890 tx=0x7f9478001170 comp rx=0 tx=0).stop 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 shutdown_connections 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f94881f8020 0x7f948815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94880c9780 0x7f94880b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f946806c980 0x7f946806ee40 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 --2- 172.21.15.70:0/1123048811 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94740039b0 0x7f94881535c0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.457 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 >> 172.21.15.70:0/1123048811 conn(0x7f948808bf60 msgr2=0x7f948806e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 shutdown_connections 2024-04-15T15:17:06.458 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.454+0000 7f948d2df640 1 -- 172.21.15.70:0/1123048811 wait complete. 2024-04-15T15:17:06.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:06.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.570+0000 7f8c1ba74640 1 Processor -- start 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.570+0000 7f8c1ba74640 1 -- start start 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1ba74640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c1415ec10 0x7f8c140b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1ba74640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f8c140b53c0 0x7f8c140b5870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1ba74640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c14067e30 con 0x7f8c1415e830 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1ba74640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f8c140b3c10 con 0x7f8c1415ec10 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1ba74640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f8c140b3ef0 con 0x7f8c140b53c0 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1900f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f8c1415ec10 0x7f8c140b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.582 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c19810640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8c1415e830 0x7f8c14067720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60380/0 (socket says 172.21.15.70:60380) 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c19810640 1 -- 172.21.15.70:0/2807828294 learned_addr learned my addr 172.21.15.70:0/2807828294 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1a011640 1 --2- 172.21.15.70:0/2807828294 >> v2:172.21.15.96:3300/0 conn(0x7f8c140b53c0 0x7f8c140b5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1900f640 1 -- 172.21.15.70:0/2807828294 >> v2:172.21.15.96:3300/0 conn(0x7f8c140b53c0 msgr2=0x7f8c140b5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1900f640 1 --2- 172.21.15.70:0/2807828294 >> v2:172.21.15.96:3300/0 conn(0x7f8c140b53c0 0x7f8c140b5870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1900f640 1 -- 172.21.15.70:0/2807828294 >> v1:172.21.15.70:6789/0 conn(0x7f8c1415e830 legacy=0x7f8c14067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.571+0000 7f8c1900f640 1 -- 172.21.15.70:0/2807828294 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c140b41d0 con 0x7f8c1415ec10 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.572+0000 7f8c1900f640 1 --2- 172.21.15.70:0/2807828294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c1415ec10 0x7f8c140b4cb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8c140b51f0 tx=0x7f8c0402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c1cbe97760c642 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.583 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.572+0000 7f8c1880e640 1 -- 172.21.15.70:0/2807828294 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c040315a0 con 0x7f8c1415ec10 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.572+0000 7f8c1880e640 1 -- 172.21.15.70:0/2807828294 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c04037070 con 0x7f8c1415ec10 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.572+0000 7f8c1880e640 1 -- 172.21.15.70:0/2807828294 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c04031960 con 0x7f8c1415ec10 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 -- 172.21.15.70:0/2807828294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c1415ec10 msgr2=0x7f8c140b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/2807828294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c1415ec10 0x7f8c140b4cb0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8c140b51f0 tx=0x7f8c0402b840 comp rx=0 tx=0).stop 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 -- 172.21.15.70:0/2807828294 shutdown_connections 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/2807828294 >> v2:172.21.15.96:3300/0 conn(0x7f8c140b53c0 0x7f8c140b5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.584 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/2807828294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c1415ec10 0x7f8c140b4cb0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 -- 172.21.15.70:0/2807828294 >> 172.21.15.70:0/2807828294 conn(0x7f8c14073360 msgr2=0x7f8c1406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 -- 172.21.15.70:0/2807828294 shutdown_connections 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.573+0000 7f8c1ba74640 1 -- 172.21.15.70:0/2807828294 wait complete. 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.574+0000 7f8c1ba74640 1 Processor -- start 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.574+0000 7f8c1ba74640 1 -- start start 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 0x7f8c141f4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c1415e830 0x7f8c140c5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.585 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1415ff60 con 0x7f8c140c6020 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f8c140b4790 con 0x7f8c140b53c0 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1ba74640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f8c141600e0 con 0x7f8c1415e830 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45414/0 (socket says 172.21.15.70:45414) 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 -- 172.21.15.70:0/954646542 learned_addr learned my addr 172.21.15.70:0/954646542 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1900f640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c1415e830 0x7f8c140c5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c19810640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 0x7f8c141f4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 -- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c1415e830 msgr2=0x7f8c140c5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c1415e830 0x7f8c140c5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.586 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 -- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 msgr2=0x7f8c141f4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 0x7f8c141f4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 -- 172.21.15.70:0/954646542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c141e7aa0 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c19810640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 0x7f8c141f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c1a011640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007e70 tx=0x7f8c0c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c014620 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.575+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c019070 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.576+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c141e77c0 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.576+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c014a20 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.576+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c14148620 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.576+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c140689e0 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.577+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f8c0c014bc0 con 0x7f8c140c6020 2024-04-15T15:17:06.587 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.577+0000 7f8c0a7fc640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 0x7f8bf806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.577+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f8c0c08c6e0 con 0x7f8c140c6020 2024-04-15T15:17:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.577+0000 7f8c19810640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 0x7f8bf806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.578+0000 7f8c19810640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 0x7f8bf806ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000f60 tx=0x7f8c10009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:06.588 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.582+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f8c0c05a920 con 0x7f8c140c6020 2024-04-15T15:17:06.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.770+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8c1406ff10 con 0x7f8c140c6020 2024-04-15T15:17:06.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.770+0000 7f8c0a7fc640 1 -- 172.21.15.70:0/954646542 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f8c0c05a220 con 0x7f8c140c6020 2024-04-15T15:17:06.772 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6835/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:06.772 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6833/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:06.772 INFO:teuthology.orchestra.run.smithi070.stderr:listed 2 entries 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 msgr2=0x7f8bf806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 0x7f8bf806ee40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8c10000f60 tx=0x7f8c10009040 comp rx=0 tx=0).stop 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 msgr2=0x7f8c141e6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c007e70 tx=0x7f8c0c00e4b0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 shutdown_connections 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f8bf806c980 0x7f8bf806ee40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8c140c6020 0x7f8c141e6b70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f8c1415e830 0x7f8c140c5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 --2- 172.21.15.70:0/954646542 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f8c140b53c0 0x7f8c141f4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:06.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 >> 172.21.15.70:0/954646542 conn(0x7f8c14073360 msgr2=0x7f8c1415b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:06.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 shutdown_connections 2024-04-15T15:17:06.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:06.772+0000 7f8c1ba74640 1 -- 172.21.15.70:0/954646542 wait complete. 2024-04-15T15:17:06.785 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:17:06.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:06.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-15T15:17:06.814 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi070.stderr:ceph-fuse[55292]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:06.815 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:06.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:06.816 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-15T15:17:06.983 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:06.983 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:17:06.983 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:07.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:07.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.126+0000 7ff9a4f3d640 1 Processor -- start 2024-04-15T15:17:07.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.126+0000 7ff9a4f3d640 1 -- start start 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7ff9a015d090 0x7ff9a0160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a0067720 0x7ff9a006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9a006ea30 0x7ff9a00cd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a0160130 con 0x7ff9a006ea30 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7ff9a01610c0 con 0x7ff9a015d090 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff9a4f3d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a015d660 con 0x7ff9a0067720 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9a006ea30 0x7ff9a00cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9a006ea30 0x7ff9a00cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45424/0 (socket says 172.21.15.70:45424) 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 -- 172.21.15.70:0/4179205357 learned_addr learned my addr 172.21.15.70:0/4179205357 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:07.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 -- 172.21.15.70:0/4179205357 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a0067720 msgr2=0x7ff9a006e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99dd9b640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a0067720 0x7ff9a006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99e59c640 1 --2- 172.21.15.70:0/4179205357 >> v2:172.21.15.96:3300/0 conn(0x7ff9a015d090 0x7ff9a0160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a0067720 0x7ff9a006e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 -- 172.21.15.70:0/4179205357 >> v2:172.21.15.96:3300/0 conn(0x7ff9a015d090 msgr2=0x7ff9a0160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 --2- 172.21.15.70:0/4179205357 >> v2:172.21.15.96:3300/0 conn(0x7ff9a015d090 0x7ff9a0160b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.128+0000 7ff99ed9d640 1 -- 172.21.15.70:0/4179205357 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a015d940 con 0x7ff9a006ea30 2024-04-15T15:17:07.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff99ed9d640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a006ea30 0x7ff9a00cd550 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0069f60 tx=0x7ff99002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=207e227af01b4b89 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff99d59a640 1 -- 172.21.15.70:0/4179205357 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff99002e070 con 0x7ff9a006ea30 2024-04-15T15:17:07.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff99d59a640 1 -- 172.21.15.70:0/4179205357 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff990037070 con 0x7ff9a006ea30 2024-04-15T15:17:07.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff99d59a640 1 -- 172.21.15.70:0/4179205357 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff990031a00 con 0x7ff9a006ea30 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/4179205357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a006ea30 msgr2=0x7ff9a00cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.129+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a006ea30 0x7ff9a00cd550 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0069f60 tx=0x7ff99002b880 comp rx=0 tx=0).stop 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/4179205357 shutdown_connections 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a006ea30 0x7ff9a00cd550 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/4179205357 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a0067720 0x7ff9a006e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/4179205357 >> v2:172.21.15.96:3300/0 conn(0x7ff9a015d090 0x7ff9a0160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/4179205357 >> 172.21.15.70:0/4179205357 conn(0x7ff9a0073360 msgr2=0x7ff9a006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/4179205357 shutdown_connections 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.130+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/4179205357 wait complete. 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.131+0000 7ff9a4f3d640 1 Processor -- start 2024-04-15T15:17:07.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.131+0000 7ff9a4f3d640 1 -- start start 2024-04-15T15:17:07.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a006ea30 0x7ff9a00abc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a015d090 0x7ff9a00ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a00b0460 con 0x7ff9a0067720 2024-04-15T15:17:07.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9a0160660 con 0x7ff9a006ea30 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff9a00b05e0 con 0x7ff9a015d090 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45432/0 (socket says 172.21.15.70:45432) 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 -- 172.21.15.70:0/61690144 learned_addr learned my addr 172.21.15.70:0/61690144 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 -- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a015d090 msgr2=0x7ff9a00ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a015d090 0x7ff9a00ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 -- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a006ea30 msgr2=0x7ff9a00abc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a006ea30 0x7ff9a00abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 -- 172.21.15.70:0/61690144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9a00aee10 con 0x7ff9a0067720 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff99e59c640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c7f0 tx=0x7ff99800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998015070 con 0x7ff9a0067720 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff998004220 con 0x7ff9a0067720 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9a00af0f0 con 0x7ff9a0067720 2024-04-15T15:17:07.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff998007620 con 0x7ff9a0067720 2024-04-15T15:17:07.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.132+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9a00af6b0 con 0x7ff9a0067720 2024-04-15T15:17:07.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.133+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9a00689e0 con 0x7ff9a0067720 2024-04-15T15:17:07.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.134+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff9980079d0 con 0x7ff9a0067720 2024-04-15T15:17:07.137 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.134+0000 7ff98effd640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 0x7ff98006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.134+0000 7ff99dd9b640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 0x7ff98006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.134+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7ff99808be60 con 0x7ff9a0067720 2024-04-15T15:17:07.138 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.135+0000 7ff99dd9b640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 0x7ff98006ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff994007130 tx=0x7ff994007310 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.140 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.139+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff99805a020 con 0x7ff9a0067720 2024-04-15T15:17:07.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.313+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff9a00ccff0 con 0x7ff9a0067720 2024-04-15T15:17:07.317 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.315+0000 7ff98effd640 1 -- 172.21.15.70:0/61690144 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7ff998059920 con 0x7ff9a0067720 2024-04-15T15:17:07.317 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:07.317 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.316+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 msgr2=0x7ff98006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.318 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.316+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 0x7ff98006ee40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff994007130 tx=0x7ff994007310 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.316+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 msgr2=0x7ff9a014b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.316+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff99800c7f0 tx=0x7ff99800ccb0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 shutdown_connections 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9a015d090 0x7ff9a00ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9a006ea30 0x7ff9a00abc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff98006c980 0x7ff98006ee40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 --2- 172.21.15.70:0/61690144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9a0067720 0x7ff9a014b300 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 >> 172.21.15.70:0/61690144 conn(0x7ff9a0073360 msgr2=0x7ff9a00b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 shutdown_connections 2024-04-15T15:17:07.319 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.317+0000 7ff9a4f3d640 1 -- 172.21.15.70:0/61690144 wait complete. 2024-04-15T15:17:07.330 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:07.330 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:07.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.433+0000 7fd28fd1c640 1 Processor -- start 2024-04-15T15:17:07.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.433+0000 7fd28fd1c640 1 -- start start 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28fd1c640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd2880d0690 0x7fd2880d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28fd1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd28815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28fd1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2880d0fb0 con 0x7fd28815ef70 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28fd1c640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd28815cb50 con 0x7fd2880d0690 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28fd1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd28815bfd0 con 0x7fd2880b3160 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28e2b9640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd28815ef70 0x7fd28815f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60396/0 (socket says 172.21.15.70:60396) 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28e2b9640 1 -- 172.21.15.70:0/1829712890 learned_addr learned my addr 172.21.15.70:0/1829712890 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28d2b7640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd28815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28dab8640 1 --2- 172.21.15.70:0/1829712890 >> v2:172.21.15.96:3300/0 conn(0x7fd2880d0690 0x7fd2880d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.434+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 124135581 0 0) 0x7fd2880d0fb0 con 0x7fd28815ef70 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd270003520 con 0x7fd28815ef70 2024-04-15T15:17:07.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3375092435 0 0) 0x7fd270003520 con 0x7fd28815ef70 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 msgr2=0x7fd28815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd28815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 >> v2:172.21.15.96:3300/0 conn(0x7fd2880d0690 msgr2=0x7fd2880d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 --2- 172.21.15.70:0/1829712890 >> v2:172.21.15.96:3300/0 conn(0x7fd2880d0690 0x7fd2880d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28815c2b0 con 0x7fd28815ef70 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fd280001670 con 0x7fd28815ef70 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 >> v1:172.21.15.70:6789/0 conn(0x7fd28815ef70 legacy=0x7fd28815f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd270003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd270004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd270004d40 0x7fd2700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd28815cb50 con 0x7fd2700039b0 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd28815bfd0 con 0x7fd270004d40 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd27000b9c0 con 0x7fd2880b3160 2024-04-15T15:17:07.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd270003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28e2b9640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd270004d40 0x7fd2700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28d2b7640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd270004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 -- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 msgr2=0x7fd270004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd2880b3160 0x7fd270004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 -- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd270004d40 msgr2=0x7fd2700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd270004d40 0x7fd2700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28dab8640 1 -- 172.21.15.70:0/1829712890 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd270003520 con 0x7fd2700039b0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.435+0000 7fd28e2b9640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd270004d40 0x7fd2700051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.436+0000 7fd28dab8640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd270003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd278009fc0 tx=0x7fd270005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c391978008b7bf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.436+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278039070 con 0x7fd2700039b0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.436+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27802cc60 con 0x7fd2700039b0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.436+0000 7fd28cab6640 1 -- 172.21.15.70:0/1829712890 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278009c10 con 0x7fd2700039b0 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 -- 172.21.15.70:0/1829712890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 msgr2=0x7fd270003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/1829712890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd270003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd278009fc0 tx=0x7fd270005830 comp rx=0 tx=0).stop 2024-04-15T15:17:07.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28e2b9640 1 -- 172.21.15.70:0/1829712890 reap_dead start 2024-04-15T15:17:07.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 -- 172.21.15.70:0/1829712890 shutdown_connections 2024-04-15T15:17:07.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 -- 172.21.15.70:0/1829712890 >> 172.21.15.70:0/1829712890 conn(0x7fd28806be00 msgr2=0x7fd28806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 -- 172.21.15.70:0/1829712890 shutdown_connections 2024-04-15T15:17:07.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.437+0000 7fd28fd1c640 1 -- 172.21.15.70:0/1829712890 wait complete. 2024-04-15T15:17:07.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.438+0000 7fd28fd1c640 1 Processor -- start 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.438+0000 7fd28fd1c640 1 -- start start 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd28809a250 0x7fd288146770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd28809d790 0x7fd28809dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2880b4fb0 con 0x7fd2700039b0 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd28815c870 con 0x7fd28809d790 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd28815fae0 con 0x7fd28809a250 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45458/0 (socket says 172.21.15.70:45458) 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 -- 172.21.15.70:0/2709361461 learned_addr learned my addr 172.21.15.70:0/2709361461 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28d2b7640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd28809a250 0x7fd288146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28e2b9640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd28809d790 0x7fd28809dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 -- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd28809a250 msgr2=0x7fd288146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd28809a250 0x7fd288146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 -- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd28809d790 msgr2=0x7fd28809dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd28809d790 0x7fd28809dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 -- 172.21.15.70:0/2709361461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd28809a7c0 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28dab8640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd278037040 tx=0x7fd278009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278039050 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27803a040 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd28809aaa0 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278009c10 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.439+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd28809b060 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.440+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2880689e0 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.441+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd278047030 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.441+0000 7fd2867fc640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 0x7fd27406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.441+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd2780ae000 con 0x7fd2700039b0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.441+0000 7fd28d2b7640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 0x7fd27406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.442+0000 7fd28d2b7640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 0x7fd27406ee70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2880af720 tx=0x7fd27c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.446+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd27804e060 con 0x7fd2700039b0 2024-04-15T15:17:07.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.620+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd2880739e0 con 0x7fd2700039b0 2024-04-15T15:17:07.622 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.620+0000 7fd2867fc640 1 -- 172.21.15.70:0/2709361461 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7fd27807c2d0 con 0x7fd2700039b0 2024-04-15T15:17:07.622 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 msgr2=0x7fd27406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 0x7fd27406ee70 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd2880af720 tx=0x7fd27c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 msgr2=0x7fd2880aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd278037040 tx=0x7fd278009750 comp rx=0 tx=0).stop 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 shutdown_connections 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd28809d790 0x7fd28809dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd28809a250 0x7fd288146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd27406c9b0 0x7fd27406ee70 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 --2- 172.21.15.70:0/2709361461 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2700039b0 0x7fd2880aeae0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 >> 172.21.15.70:0/2709361461 conn(0x7fd28806be00 msgr2=0x7fd28815d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 shutdown_connections 2024-04-15T15:17:07.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.622+0000 7fd28fd1c640 1 -- 172.21.15.70:0/2709361461 wait complete. 2024-04-15T15:17:07.635 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:07.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.740+0000 7f9417d00640 1 Processor -- start 2024-04-15T15:17:07.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.740+0000 7f9417d00640 1 -- start start 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9417d00640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9417d00640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94100b1d30 0x7f94100af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9417d00640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94100b10d0 con 0x7f94100b4350 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9417d00640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f941015f090 con 0x7f94100b1d30 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9417d00640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f94100cdf80 con 0x7f941015ecb0 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:53712/0 (socket says 172.21.15.70:53712) 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f941629d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f94100b1d30 0x7f94100af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 -- 172.21.15.70:0/376298406 learned_addr learned my addr 172.21.15.70:0/376298406 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 -- 172.21.15.70:0/376298406 >> v2:172.21.15.96:3300/0 conn(0x7f94100b1d30 msgr2=0x7f94100af340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 --2- 172.21.15.70:0/376298406 >> v2:172.21.15.96:3300/0 conn(0x7f94100b1d30 0x7f94100af340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 -- 172.21.15.70:0/376298406 >> v1:172.21.15.96:6789/0 conn(0x7f94100b4350 legacy=0x7f94100b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.741+0000 7f9415a9c640 1 -- 172.21.15.70:0/376298406 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94100af920 con 0x7f941015ecb0 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.742+0000 7f9415a9c640 1 --2- 172.21.15.70:0/376298406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ba732010cec5be6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.742+0000 7f9414a9a640 1 -- 172.21.15.70:0/376298406 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c02cb90 con 0x7f941015ecb0 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.742+0000 7f9414a9a640 1 -- 172.21.15.70:0/376298406 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940c02cd30 con 0x7f941015ecb0 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.742+0000 7f9414a9a640 1 -- 172.21.15.70:0/376298406 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c034610 con 0x7f941015ecb0 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 -- 172.21.15.70:0/376298406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 msgr2=0x7f94100b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 --2- 172.21.15.70:0/376298406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f940c009e50 tx=0x7f940c02bff0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 -- 172.21.15.70:0/376298406 shutdown_connections 2024-04-15T15:17:07.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 --2- 172.21.15.70:0/376298406 >> v2:172.21.15.96:3300/0 conn(0x7f94100b1d30 0x7f94100af340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 --2- 172.21.15.70:0/376298406 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f941015ecb0 0x7f94100b0b90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 -- 172.21.15.70:0/376298406 >> 172.21.15.70:0/376298406 conn(0x7f94100b6870 msgr2=0x7f94100b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 -- 172.21.15.70:0/376298406 shutdown_connections 2024-04-15T15:17:07.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.743+0000 7f9417d00640 1 -- 172.21.15.70:0/376298406 wait complete. 2024-04-15T15:17:07.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.744+0000 7f9417d00640 1 Processor -- start 2024-04-15T15:17:07.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.744+0000 7f9417d00640 1 -- start start 2024-04-15T15:17:07.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94100b4350 0x7f9410160c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 0x7f9410158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f94100b4790 con 0x7f94100b1d30 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f94100afee0 con 0x7f941015ecb0 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9417d00640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f94100b4910 con 0x7f94100b4350 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9415a9c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9415a9c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45474/0 (socket says 172.21.15.70:45474) 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9415a9c640 1 -- 172.21.15.70:0/3712210922 learned_addr learned my addr 172.21.15.70:0/3712210922 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f941529b640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94100b4350 0x7f9410160c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f941629d640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 0x7f9410158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.745+0000 7f9415a9c640 1 -- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94100b4350 msgr2=0x7f9410160c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9415a9c640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94100b4350 0x7f9410160c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9415a9c640 1 -- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 msgr2=0x7f9410158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9415a9c640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 0x7f9410158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9415a9c640 1 -- 172.21.15.70:0/3712210922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f941015fb80 con 0x7f94100b1d30 2024-04-15T15:17:07.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f941629d640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 0x7f9410158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9415a9c640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f940c036040 tx=0x7f940c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c044020 con 0x7f94100b1d30 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f940c040690 con 0x7f94100b1d30 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f941015fe60 con 0x7f94100b1d30 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f940c009510 con 0x7f94100b1d30 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.746+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9410160420 con 0x7f94100b1d30 2024-04-15T15:17:07.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.747+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94100689e0 con 0x7f94100b1d30 2024-04-15T15:17:07.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.747+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f940c047020 con 0x7f94100b1d30 2024-04-15T15:17:07.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.748+0000 7f94067fc640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 0x7f93f406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:07.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.748+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f940c0ae710 con 0x7f94100b1d30 2024-04-15T15:17:07.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.748+0000 7f941529b640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 0x7f93f406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:07.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.748+0000 7f941529b640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 0x7f93f406ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94100cd310 tx=0x7f9400009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:07.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.752+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f940c04e030 con 0x7f94100b1d30 2024-04-15T15:17:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.927+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f941006c7c0 con 0x7f94100b1d30 2024-04-15T15:17:07.929 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.927+0000 7f94067fc640 1 -- 172.21.15.70:0/3712210922 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f940c07c950 con 0x7f94100b1d30 2024-04-15T15:17:07.929 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 msgr2=0x7f93f406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 0x7f93f406ee40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f94100cd310 tx=0x7f9400009040 comp rx=0 tx=0).stop 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 msgr2=0x7f941015f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f940c036040 tx=0x7f940c004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 shutdown_connections 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f941015ecb0 0x7f9410158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f94100b4350 0x7f9410160c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f93f406c980 0x7f93f406ee40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.932 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 --2- 172.21.15.70:0/3712210922 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f94100b1d30 0x7f941015f210 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:07.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 >> 172.21.15.70:0/3712210922 conn(0x7f94100b6870 msgr2=0x7f94100b12a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:07.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.929+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 shutdown_connections 2024-04-15T15:17:07.933 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:07.930+0000 7f9417d00640 1 -- 172.21.15.70:0/3712210922 wait complete. 2024-04-15T15:17:07.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:08.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.046+0000 7f0cab62a640 1 Processor -- start 2024-04-15T15:17:08.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.046+0000 7f0cab62a640 1 -- start start 2024-04-15T15:17:08.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0cab62a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f0ca415d090 0x7f0ca4160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0cab62a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca4067720 0x7f0ca406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0cab62a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ca415d660 con 0x7f0ca406ea30 2024-04-15T15:17:08.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0cab62a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca4160130 con 0x7f0ca4067720 2024-04-15T15:17:08.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0cab62a640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f0ca41610c0 con 0x7f0ca415d090 2024-04-15T15:17:08.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca4067720 0x7f0ca406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f0ca4067720 0x7f0ca406e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45486/0 (socket says 172.21.15.70:45486) 2024-04-15T15:17:08.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/3299718758 learned_addr learned my addr 172.21.15.70:0/3299718758 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:08.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca93c6640 1 --2- 172.21.15.70:0/3299718758 >> v2:172.21.15.96:3300/0 conn(0x7f0ca415d090 0x7f0ca4160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.051 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/3299718758 >> v2:172.21.15.96:3300/0 conn(0x7f0ca415d090 msgr2=0x7f0ca4160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 --2- 172.21.15.70:0/3299718758 >> v2:172.21.15.96:3300/0 conn(0x7f0ca415d090 0x7f0ca4160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.047+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/3299718758 >> v1:172.21.15.70:6789/0 conn(0x7f0ca406ea30 legacy=0x7f0ca40cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.052 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.048+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/3299718758 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca415d940 con 0x7f0ca4067720 2024-04-15T15:17:08.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.048+0000 7f0ca8bc5640 1 --2- 172.21.15.70:0/3299718758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca4067720 0x7f0ca406e4f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4069f60 tx=0x7f0c9402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c790cdd8c9465a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.048+0000 7f0c9bfff640 1 -- 172.21.15.70:0/3299718758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94035020 con 0x7f0ca4067720 2024-04-15T15:17:08.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.048+0000 7f0c9bfff640 1 -- 172.21.15.70:0/3299718758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94031870 con 0x7f0ca4067720 2024-04-15T15:17:08.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.048+0000 7f0c9bfff640 1 -- 172.21.15.70:0/3299718758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94031c40 con 0x7f0ca4067720 2024-04-15T15:17:08.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 -- 172.21.15.70:0/3299718758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca4067720 msgr2=0x7f0ca406e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 --2- 172.21.15.70:0/3299718758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca4067720 0x7f0ca406e4f0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4069f60 tx=0x7f0c9402b910 comp rx=0 tx=0).stop 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 -- 172.21.15.70:0/3299718758 shutdown_connections 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 --2- 172.21.15.70:0/3299718758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca4067720 0x7f0ca406e4f0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 --2- 172.21.15.70:0/3299718758 >> v2:172.21.15.96:3300/0 conn(0x7f0ca415d090 0x7f0ca4160b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 -- 172.21.15.70:0/3299718758 >> 172.21.15.70:0/3299718758 conn(0x7f0ca4073360 msgr2=0x7f0ca406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 -- 172.21.15.70:0/3299718758 shutdown_connections 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.049+0000 7f0cab62a640 1 -- 172.21.15.70:0/3299718758 wait complete. 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.050+0000 7f0cab62a640 1 Processor -- start 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.050+0000 7f0cab62a640 1 -- start start 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 0x7f0ca4153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 0x7f0ca40a99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca40b0460 con 0x7f0ca406ea30 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4160660 con 0x7f0ca415d090 2024-04-15T15:17:08.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0cab62a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f0ca40b05e0 con 0x7f0ca4067720 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45494/0 (socket says 172.21.15.70:45494) 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/2311410844 learned_addr learned my addr 172.21.15.70:0/2311410844 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca93c6640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 0x7f0ca4153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca9bc7640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 0x7f0ca40a99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 msgr2=0x7f0ca4153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 0x7f0ca4153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 msgr2=0x7f0ca40a99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 0x7f0ca40a99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca8bc5640 1 -- 172.21.15.70:0/2311410844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca4158ba0 con 0x7f0ca406ea30 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca93c6640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 0x7f0ca4153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.051+0000 7f0ca9bc7640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 0x7f0ca40a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0ca8bc5640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0c9402b950 tx=0x7f0c940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.058 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94040070 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c94030d00 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca4158e80 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94035040 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.052+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca4159720 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.053+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca40698a0 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.053+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f0c9404b060 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.054+0000 7f0c99ffb640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 0x7f0c8806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.054+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f0c940adb80 con 0x7f0ca406ea30 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.054+0000 7f0ca93c6640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 0x7f0c8806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.059 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.054+0000 7f0ca93c6640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 0x7f0c8806ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40cf730 tx=0x7f0ca00039f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.058+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0ca40698a0 con 0x7f0ca406ea30 2024-04-15T15:17:08.237 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.235+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0ca40689e0 con 0x7f0ca406ea30 2024-04-15T15:17:08.238 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.236+0000 7f0c99ffb640 1 -- 172.21.15.70:0/2311410844 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f0c9407be50 con 0x7f0ca406ea30 2024-04-15T15:17:08.238 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 msgr2=0x7f0c8806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 0x7f0c8806ee40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40cf730 tx=0x7f0ca00039f0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 msgr2=0x7f0ca4158150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f0c9402b950 tx=0x7f0c940046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 shutdown_connections 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0ca415d090 0x7f0ca40a99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f0c8806c980 0x7f0c8806ee40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ca406ea30 0x7f0ca4158150 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 --2- 172.21.15.70:0/2311410844 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0ca4067720 0x7f0ca4153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 >> 172.21.15.70:0/2311410844 conn(0x7f0ca4073360 msgr2=0x7f0ca406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 shutdown_connections 2024-04-15T15:17:08.241 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.238+0000 7f0cab62a640 1 -- 172.21.15.70:0/2311410844 wait complete. 2024-04-15T15:17:08.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:08.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.353+0000 7fc6075bd640 1 Processor -- start 2024-04-15T15:17:08.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.353+0000 7fc6075bd640 1 -- start start 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc6075bd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc6000d0690 0x7fc6000d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc6075bd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6000d0fb0 con 0x7fc6000ae280 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc6075bd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc60015cb50 con 0x7fc6000b3160 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc6075bd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc60015bda0 con 0x7fc6000d0690 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc605b5a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc6000ae280 0x7fc6000ae710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60414/0 (socket says 172.21.15.70:60414) 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc605b5a640 1 -- 172.21.15.70:0/1245613290 learned_addr learned my addr 172.21.15.70:0/1245613290 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> v2:172.21.15.96:3300/0 conn(0x7fc6000d0690 0x7fc6000d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 245971498 0 0) 0x7fc6000d0fb0 con 0x7fc6000ae280 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.354+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5ec003520 con 0x7fc6000ae280 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3771731258 0 0) 0x7fc60015cb50 con 0x7fc6000b3160 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6000d0fb0 con 0x7fc6000b3160 2024-04-15T15:17:08.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2470058171 0 0) 0x7fc5ec003520 con 0x7fc6000ae280 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 >> v2:172.21.15.96:3300/0 conn(0x7fc6000d0690 msgr2=0x7fc6000d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 --2- 172.21.15.70:0/1245613290 >> v2:172.21.15.96:3300/0 conn(0x7fc6000d0690 0x7fc6000d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 >> v1:172.21.15.96:6789/0 conn(0x7fc6000b3160 legacy=0x7fc60015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc60015bfd0 con 0x7fc6000ae280 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fc5f80019f0 con 0x7fc6000ae280 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> v2:172.21.15.96:3300/0 conn(0x7fc6000d0690 0x7fc6000d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 >> v1:172.21.15.70:6789/0 conn(0x7fc6000ae280 legacy=0x7fc6000ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc5ec003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5ec004800 0x7fc5ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ec0051c0 0x7fc5ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec00c010 con 0x7fc5ec003b40 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc5ec00c1c0 con 0x7fc5ec0051c0 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc5ec00c3c0 con 0x7fc5ec004800 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc5ec003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605b5a640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ec0051c0 0x7fc5ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc604b58640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5ec004800 0x7fc5ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 -- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5ec004800 msgr2=0x7fc5ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc5ec004800 0x7fc5ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 -- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ec0051c0 msgr2=0x7fc5ec00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc5ec0051c0 0x7fc5ec00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605b5a640 1 -- 172.21.15.70:0/1245613290 reap_dead start 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.355+0000 7fc605359640 1 -- 172.21.15.70:0/1245613290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5ec003520 con 0x7fc5ec003b40 2024-04-15T15:17:08.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.356+0000 7fc605359640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc5ec003f40 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009c90 tx=0x7fc5ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fec17ca2a74ec9 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.356+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc039070 con 0x7fc5ec003b40 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.356+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc02c8f0 con 0x7fc5ec003b40 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.356+0000 7fc5f7fff640 1 -- 172.21.15.70:0/1245613290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc009860 con 0x7fc5ec003b40 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 -- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 msgr2=0x7fc5ec003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc5ec003f40 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009c90 tx=0x7fc5ec005830 comp rx=0 tx=0).stop 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 -- 172.21.15.70:0/1245613290 shutdown_connections 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 --2- 172.21.15.70:0/1245613290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc5ec003f40 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.362 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 -- 172.21.15.70:0/1245613290 >> 172.21.15.70:0/1245613290 conn(0x7fc60006be00 msgr2=0x7fc60006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 -- 172.21.15.70:0/1245613290 shutdown_connections 2024-04-15T15:17:08.363 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.357+0000 7fc6075bd640 1 -- 172.21.15.70:0/1245613290 wait complete. 2024-04-15T15:17:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.358+0000 7fc6075bd640 1 Processor -- start 2024-04-15T15:17:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.358+0000 7fc6075bd640 1 -- start start 2024-04-15T15:17:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 0x7fc600072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6000ce470 0x7fc6000ce8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc60015c870 con 0x7fc5ec003b40 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc60015c590 con 0x7fc600072a60 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc6000b4fb0 con 0x7fc6000ce470 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45514/0 (socket says 172.21.15.70:45514) 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 -- 172.21.15.70:0/4162772222 learned_addr learned my addr 172.21.15.70:0/4162772222 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605b5a640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6000ce470 0x7fc6000ce8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc604b58640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 0x7fc600072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 -- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6000ce470 msgr2=0x7fc6000ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6000ce470 0x7fc6000ce8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 -- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 msgr2=0x7fc600072ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 0x7fc600072ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 -- 172.21.15.70:0/4162772222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6000cf1c0 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc604b58640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 0x7fc600072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc605359640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009db0 tx=0x7fc5fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc039050 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5fc03a040 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc004460 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.359+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6000cf4a0 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.360+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc60015d360 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.361+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc60015ce70 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.361+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc5fc034880 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.361+0000 7fc5f5ffb640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 0x7fc5e406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.362+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc5fc0af520 con 0x7fc5ec003b40 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.362+0000 7fc604b58640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 0x7fc5e406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.362+0000 7fc604b58640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 0x7fc5e406ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.366+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc5fc07d6d0 con 0x7fc5ec003b40 2024-04-15T15:17:08.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.552+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc6000689e0 con 0x7fc5ec003b40 2024-04-15T15:17:08.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.553+0000 7fc5f5ffb640 1 -- 172.21.15.70:0/4162772222 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7fc5fc07cfd0 con 0x7fc5ec003b40 2024-04-15T15:17:08.555 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6835/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:08.555 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6833/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:08.555 INFO:teuthology.orchestra.run.smithi070.stderr:listed 2 entries 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 msgr2=0x7fc5e406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 0x7fc5e406ee40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc5f0000c00 tx=0x7fc5f0009040 comp rx=0 tx=0).stop 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 msgr2=0x7fc600099820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc009db0 tx=0x7fc5fc004730 comp rx=0 tx=0).stop 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 shutdown_connections 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc6000ce470 0x7fc6000ce8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc600072a60 0x7fc600072ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc5e406c980 0x7fc5e406ee40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 --2- 172.21.15.70:0/4162772222 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc5ec003b40 0x7fc600099820 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 >> 172.21.15.70:0/4162772222 conn(0x7fc60006be00 msgr2=0x7fc600156ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 shutdown_connections 2024-04-15T15:17:08.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.555+0000 7fc6075bd640 1 -- 172.21.15.70:0/4162772222 wait complete. 2024-04-15T15:17:08.568 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-15T15:17:08.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:08.568 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-15T15:17:08.601 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi096.stderr:ceph-fuse[45656]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:08.603 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:08.603 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:08.603 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-15T15:17:08.775 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:08.775 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:17:08.775 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:08.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:08.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.933+0000 7f4ece1db640 1 Processor -- start 2024-04-15T15:17:08.935 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.933+0000 7f4ece1db640 1 -- start start 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ece1db640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4ec80d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ece1db640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec80d0fb0 con 0x7f4ec80b3160 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ece1db640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ec815cb50 con 0x7f4ec80ae280 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ece1db640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec815bda0 con 0x7f4ec80d0690 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4ec80b3160 0x7f4ec815b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60428/0 (socket says 172.21.15.70:60428) 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec6ffd640 1 -- 172.21.15.70:0/2626377758 learned_addr learned my addr 172.21.15.70:0/2626377758 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec77fe640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4ec80d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972431159 0 0) 0x7f4ec80d0fb0 con 0x7f4ec80b3160 2024-04-15T15:17:08.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4eb0003520 con 0x7f4ec80b3160 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4228624137 0 0) 0x7f4eb0003520 con 0x7f4ec80b3160 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 msgr2=0x7f4ec80d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4ec80d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 >> v1:172.21.15.96:6789/0 conn(0x7f4ec80ae280 legacy=0x7f4ec80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec815bfd0 con 0x7f4ec80b3160 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4ebc001670 con 0x7f4ec80b3160 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 >> v1:172.21.15.70:6789/0 conn(0x7f4ec80b3160 legacy=0x7f4ec815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.934+0000 7f4ec67fc640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4eb00039e0 0x7f4eb0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec67fc640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4eb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec67fc640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0004d40 0x7f4eb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb000ba70 con 0x7f4eb0004d40 2024-04-15T15:17:08.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4eb000bbf0 con 0x7f4eb00039e0 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4eb000e000 con 0x7f4ec80d0690 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0004d40 0x7f4eb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec77fe640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4eb00039e0 0x7f4eb0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec6ffd640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4eb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 -- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 msgr2=0x7f4eb0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4ec80d0690 0x7f4eb0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 -- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4eb00039e0 msgr2=0x7f4eb0003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4eb00039e0 0x7f4eb0003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.935+0000 7f4ec7fff640 1 -- 172.21.15.70:0/2626377758 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4eb0003520 con 0x7f4eb0004d40 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ec7fff640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0004d40 0x7f4eb00051c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0001e20 tx=0x7f4eb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1f67a4932084eb server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec0001a70 con 0x7f4eb0004d40 2024-04-15T15:17:08.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec0003b70 con 0x7f4eb0004d40 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ec67fc640 1 -- 172.21.15.70:0/2626377758 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec00050d0 con 0x7f4eb0004d40 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 -- 172.21.15.70:0/2626377758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0004d40 msgr2=0x7f4eb00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 --2- 172.21.15.70:0/2626377758 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4eb0004d40 0x7f4eb00051c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f4ec0001e20 tx=0x7f4eb0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ec7fff640 1 -- 172.21.15.70:0/2626377758 reap_dead start 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 -- 172.21.15.70:0/2626377758 shutdown_connections 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 -- 172.21.15.70:0/2626377758 >> 172.21.15.70:0/2626377758 conn(0x7f4ec806be00 msgr2=0x7f4ec806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 -- 172.21.15.70:0/2626377758 shutdown_connections 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.936+0000 7f4ece1db640 1 -- 172.21.15.70:0/2626377758 wait complete. 2024-04-15T15:17:08.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.937+0000 7f4ece1db640 1 Processor -- start 2024-04-15T15:17:08.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 -- start start 2024-04-15T15:17:08.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4eb0004d40 0x7f4ec8151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ec809c4e0 0x7f4ec809c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec815c870 con 0x7f4ec81e5f90 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4ec815c590 con 0x7f4ec809c4e0 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ece1db640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4ec80b4fb0 con 0x7f4eb0004d40 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.938+0000 7f4ec6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45522/0 (socket says 172.21.15.70:45522) 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 -- 172.21.15.70:0/3758572604 learned_addr learned my addr 172.21.15.70:0/3758572604 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec7fff640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ec809c4e0 0x7f4ec809c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec77fe640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4eb0004d40 0x7f4ec8151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 -- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4eb0004d40 msgr2=0x7f4ec8151030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4eb0004d40 0x7f4ec8151030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 -- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ec809c4e0 msgr2=0x7f4ec809c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ec809c4e0 0x7f4ec809c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 -- 172.21.15.70:0/3758572604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ec809d250 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ec6ffd640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc007290 tx=0x7f4ebc007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00f960 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ebc018070 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ec809d530 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc00fd40 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.939+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ec809daf0 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.940+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ec809de20 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.941+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4ebc01d460 con 0x7f4ec81e5f90 2024-04-15T15:17:08.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.941+0000 7f4eaffff640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 0x7f4eb406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:08.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.941+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f4ebc08bfa0 con 0x7f4ec81e5f90 2024-04-15T15:17:08.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.941+0000 7f4ec77fe640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 0x7f4eb406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:08.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.942+0000 7f4ec77fe640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 0x7f4eb406ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009b00 tx=0x7f4eb8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:08.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:08.945+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4ebc05a1e0 con 0x7f4ec81e5f90 2024-04-15T15:17:09.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.123+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ec80689e0 con 0x7f4ec81e5f90 2024-04-15T15:17:09.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.124+0000 7f4eaffff640 1 -- 172.21.15.70:0/3758572604 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7f4ebc059ae0 con 0x7f4ec81e5f90 2024-04-15T15:17:09.126 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:09.127 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.126+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 msgr2=0x7f4eb406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.126+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 0x7f4eb406ee40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009b00 tx=0x7f4eb8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.126+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 msgr2=0x7f4ec81e6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.126+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc007290 tx=0x7f4ebc007750 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 shutdown_connections 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4ec809c4e0 0x7f4ec809c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4eb406c980 0x7f4eb406ee40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ec81e5f90 0x7f4ec81e6410 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 --2- 172.21.15.70:0/3758572604 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4eb0004d40 0x7f4ec8151030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 >> 172.21.15.70:0/3758572604 conn(0x7f4ec806be00 msgr2=0x7f4ec806dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 shutdown_connections 2024-04-15T15:17:09.129 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.127+0000 7f4ece1db640 1 -- 172.21.15.70:0/3758572604 wait complete. 2024-04-15T15:17:09.140 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:09.140 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.244+0000 7f262ccd9640 1 Processor -- start 2024-04-15T15:17:09.247 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.244+0000 7f262ccd9640 1 -- start start 2024-04-15T15:17:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f262ccd9640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26280af290 0x7f2628166300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.248 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f262ccd9640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262815cb50 con 0x7f26280d0690 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f262ccd9640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f262815bfd0 con 0x7f26280b3160 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f262ccd9640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f262815bda0 con 0x7f26280af290 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f2626d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26280af290 0x7f2628166300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.245+0000 7f2626d9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f26280af290 0x7f2628166300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45532/0 (socket says 172.21.15.70:45532) 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.246+0000 7f2626d9d640 1 -- 172.21.15.70:0/1318362575 learned_addr learned my addr 172.21.15.70:0/1318362575 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.246+0000 7f2626d9d640 1 -- 172.21.15.70:0/1318362575 >> v1:172.21.15.96:6789/0 conn(0x7f26280b3160 legacy=0x7f262815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.246+0000 7f2626d9d640 1 -- 172.21.15.70:0/1318362575 >> v1:172.21.15.70:6789/0 conn(0x7f26280d0690 legacy=0x7f26280d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.246+0000 7f2626d9d640 1 -- 172.21.15.70:0/1318362575 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f262815c2b0 con 0x7f26280af290 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.246+0000 7f2626d9d640 1 --2- 172.21.15.70:0/1318362575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280af290 0x7f2628166300 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f26200098a0 tx=0x7f262002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=deada1df38099e02 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262559a640 1 -- 172.21.15.70:0/1318362575 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262002f830 con 0x7f26280af290 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262559a640 1 -- 172.21.15.70:0/1318362575 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f262002f9d0 con 0x7f26280af290 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262559a640 1 -- 172.21.15.70:0/1318362575 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262002fda0 con 0x7f26280af290 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 -- 172.21.15.70:0/1318362575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280af290 msgr2=0x7f2628166300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.249 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1318362575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280af290 0x7f2628166300 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f26200098a0 tx=0x7f262002ba10 comp rx=0 tx=0).stop 2024-04-15T15:17:09.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 -- 172.21.15.70:0/1318362575 shutdown_connections 2024-04-15T15:17:09.250 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1318362575 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280af290 0x7f2628166300 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 -- 172.21.15.70:0/1318362575 >> 172.21.15.70:0/1318362575 conn(0x7f262806be00 msgr2=0x7f262806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 -- 172.21.15.70:0/1318362575 shutdown_connections 2024-04-15T15:17:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.247+0000 7f262ccd9640 1 -- 172.21.15.70:0/1318362575 wait complete. 2024-04-15T15:17:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 Processor -- start 2024-04-15T15:17:09.251 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 -- start start 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26280af290 0x7f2628148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f26280b3160 0x7f26281533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f26280b4fb0 con 0x7f26280d0690 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f262815c870 con 0x7f26280af290 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f262ccd9640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f262806dd30 con 0x7f26280b3160 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f2626d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45546/0 (socket says 172.21.15.70:45546) 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.249+0000 7f2626d9d640 1 -- 172.21.15.70:0/1496142066 learned_addr learned my addr 172.21.15.70:0/1496142066 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 -- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f26280b3160 msgr2=0x7f26281533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.253 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f262659c640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26280af290 0x7f2628148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2625d9b640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f26280b3160 0x7f26281533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f26280b3160 0x7f26281533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 -- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26280af290 msgr2=0x7f2628148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26280af290 0x7f2628148670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 -- 172.21.15.70:0/1496142066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2628148bb0 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2626d9d640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f26200098a0 tx=0x7f2620004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620037dd0 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2620046070 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2628153930 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2620041640 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.250+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2628164910 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.251+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26280689e0 con 0x7f26280d0690 2024-04-15T15:17:09.254 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.251+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f2620040070 con 0x7f26280d0690 2024-04-15T15:17:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.252+0000 7f2616ffd640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 0x7f260806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.252+0000 7f262659c640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 0x7f260806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.252+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f26200ae7e0 con 0x7f26280d0690 2024-04-15T15:17:09.255 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.252+0000 7f262659c640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 0x7f260806ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2618007fd0 tx=0x7f2618009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.258 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.256+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2628148bb0 con 0x7f26280d0690 2024-04-15T15:17:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.431+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f262806c2e0 con 0x7f26280d0690 2024-04-15T15:17:09.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.432+0000 7f2616ffd640 1 -- 172.21.15.70:0/1496142066 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f2620033030 con 0x7f26280d0690 2024-04-15T15:17:09.434 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 msgr2=0x7f260806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 0x7f260806ee40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2618007fd0 tx=0x7f2618009040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 msgr2=0x7f26280ccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f26200098a0 tx=0x7f2620004730 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 shutdown_connections 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f260806c980 0x7f260806ee40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f26280d0690 0x7f26280ccc80 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f26280b3160 0x7f26281533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 --2- 172.21.15.70:0/1496142066 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f26280af290 0x7f2628148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 >> 172.21.15.70:0/1496142066 conn(0x7f262806be00 msgr2=0x7f26280726e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 shutdown_connections 2024-04-15T15:17:09.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.434+0000 7f262ccd9640 1 -- 172.21.15.70:0/1496142066 wait complete. 2024-04-15T15:17:09.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.551+0000 7f61ba3a8640 1 Processor -- start 2024-04-15T15:17:09.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.551+0000 7f61ba3a8640 1 -- start start 2024-04-15T15:17:09.554 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61ba3a8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f61b415ecb0 0x7f61b40b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61ba3a8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61b40b1d30 0x7f61b40af340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61ba3a8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b40b10d0 con 0x7f61b40b4350 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61ba3a8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f61b415f090 con 0x7f61b40b1d30 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61ba3a8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f61b40cdf80 con 0x7f61b415ecb0 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61b8945640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61b40b1d30 0x7f61b40af340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61b8945640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61b40b1d30 0x7f61b40af340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45558/0 (socket says 172.21.15.70:45558) 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.552+0000 7f61b8945640 1 -- 172.21.15.70:0/783871124 learned_addr learned my addr 172.21.15.70:0/783871124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.553+0000 7f61b8945640 1 -- 172.21.15.70:0/783871124 >> v2:172.21.15.96:3300/0 conn(0x7f61b415ecb0 msgr2=0x7f61b40b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.553+0000 7f61b8945640 1 --2- 172.21.15.70:0/783871124 >> v2:172.21.15.96:3300/0 conn(0x7f61b415ecb0 0x7f61b40b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.553+0000 7f61b8945640 1 -- 172.21.15.70:0/783871124 >> v1:172.21.15.96:6789/0 conn(0x7f61b40b4350 legacy=0x7f61b40b1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.553+0000 7f61b8945640 1 -- 172.21.15.70:0/783871124 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b40af920 con 0x7f61b40b1d30 2024-04-15T15:17:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.553+0000 7f61b8945640 1 --2- 172.21.15.70:0/783871124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b1d30 0x7f61b40af340 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009e50 tx=0x7f61ac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986cf546510461c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61b37fe640 1 -- 172.21.15.70:0/783871124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac02cb20 con 0x7f61b40b1d30 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61b37fe640 1 -- 172.21.15.70:0/783871124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac02ccc0 con 0x7f61b40b1d30 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61b37fe640 1 -- 172.21.15.70:0/783871124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac034610 con 0x7f61b40b1d30 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 -- 172.21.15.70:0/783871124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b1d30 msgr2=0x7f61b40af340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/783871124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b1d30 0x7f61b40af340 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009e50 tx=0x7f61ac02bf80 comp rx=0 tx=0).stop 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 -- 172.21.15.70:0/783871124 shutdown_connections 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/783871124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b1d30 0x7f61b40af340 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/783871124 >> v2:172.21.15.96:3300/0 conn(0x7f61b415ecb0 0x7f61b40b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 -- 172.21.15.70:0/783871124 >> 172.21.15.70:0/783871124 conn(0x7f61b40b6870 msgr2=0x7f61b40b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 -- 172.21.15.70:0/783871124 shutdown_connections 2024-04-15T15:17:09.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.554+0000 7f61ba3a8640 1 -- 172.21.15.70:0/783871124 wait complete. 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.555+0000 7f61ba3a8640 1 Processor -- start 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.555+0000 7f61ba3a8640 1 -- start start 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 0x7f61b40cb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 0x7f61b40ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40b47c0 con 0x7f61b40b4350 2024-04-15T15:17:09.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40afee0 con 0x7f61b415ecb0 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61ba3a8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f61b40cc670 con 0x7f61b40b1d30 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45574/0 (socket says 172.21.15.70:45574) 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 -- 172.21.15.70:0/878224063 learned_addr learned my addr 172.21.15.70:0/878224063 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61b3fff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 0x7f61b40cb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61b8945640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 0x7f61b40ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 -- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 msgr2=0x7f61b40cb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 0x7f61b40cb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 -- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 msgr2=0x7f61b40ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 0x7f61b40ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61abfff640 1 -- 172.21.15.70:0/878224063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b40cba60 con 0x7f61b40b4350 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61b3fff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 0x7f61b40cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.556+0000 7f61b8945640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 0x7f61b40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:09.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61abfff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f619c00c410 tx=0x7f619c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c017070 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f619c0035f0 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b40cbd40 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619c003a20 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.557+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b40cc300 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.558+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b40689e0 con 0x7f61b40b4350 2024-04-15T15:17:09.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.558+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f619c01a070 con 0x7f61b40b4350 2024-04-15T15:17:09.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.559+0000 7f61b17fa640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 0x7f619406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.559+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f619c08b230 con 0x7f61b40b4350 2024-04-15T15:17:09.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.559+0000 7f61b3fff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 0x7f619406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.559+0000 7f61b3fff640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 0x7f619406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009890 tx=0x7f61a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.563+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f619c059470 con 0x7f61b40b4350 2024-04-15T15:17:09.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.737+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f61b406c7c0 con 0x7f61b40b4350 2024-04-15T15:17:09.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.737+0000 7f61b17fa640 1 -- 172.21.15.70:0/878224063 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f619c058d70 con 0x7f61b40b4350 2024-04-15T15:17:09.739 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 msgr2=0x7f619406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 0x7f619406ee40 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009890 tx=0x7f61a4008040 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 msgr2=0x7f61b4153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f619c00c410 tx=0x7f619c00c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 shutdown_connections 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61b415ecb0 0x7f61b40ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f619406c980 0x7f619406ee40 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61b40b4350 0x7f61b4153df0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 --2- 172.21.15.70:0/878224063 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f61b40b1d30 0x7f61b40cb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 >> 172.21.15.70:0/878224063 conn(0x7f61b40b6870 msgr2=0x7f61b40b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 shutdown_connections 2024-04-15T15:17:09.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.739+0000 7f61ba3a8640 1 -- 172.21.15.70:0/878224063 wait complete. 2024-04-15T15:17:09.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.856+0000 7f4946a35640 1 Processor -- start 2024-04-15T15:17:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.856+0000 7f4946a35640 1 -- start start 2024-04-15T15:17:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4940072d80 0x7f4940071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 0x7f49400af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.860 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f49400d0690 0x7f49400d0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f494015db40 con 0x7f4940072d80 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f494015cfc0 con 0x7f49400d0690 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4946a35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f494015d2a0 con 0x7f49400af3e0 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f493ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4940072d80 0x7f4940071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f493ffff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f4940072d80 0x7f4940071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45586/0 (socket says 172.21.15.70:45586) 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f493ffff640 1 -- 172.21.15.70:0/981935610 learned_addr learned my addr 172.21.15.70:0/981935610 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f493ffff640 1 -- 172.21.15.70:0/981935610 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 msgr2=0x7f49400af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4944fd2640 1 --2- 172.21.15.70:0/981935610 >> v2:172.21.15.96:3300/0 conn(0x7f49400d0690 0x7f49400d0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.857+0000 7f4937fff640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 0x7f49400af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493ffff640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 0x7f49400af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493ffff640 1 -- 172.21.15.70:0/981935610 >> v2:172.21.15.96:3300/0 conn(0x7f49400d0690 msgr2=0x7f49400d0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493ffff640 1 --2- 172.21.15.70:0/981935610 >> v2:172.21.15.96:3300/0 conn(0x7f49400d0690 0x7f49400d0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493ffff640 1 -- 172.21.15.70:0/981935610 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494015d580 con 0x7f4940072d80 2024-04-15T15:17:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f4937fff640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 0x7f49400af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493ffff640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f4940071d10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f49401600c0 tx=0x7f493002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec90cf42fbeef729 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493f7fe640 1 -- 172.21.15.70:0/981935610 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493002e070 con 0x7f4940072d80 2024-04-15T15:17:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493f7fe640 1 -- 172.21.15.70:0/981935610 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4930037070 con 0x7f4940072d80 2024-04-15T15:17:09.862 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.858+0000 7f493f7fe640 1 -- 172.21.15.70:0/981935610 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930031a00 con 0x7f4940072d80 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 -- 172.21.15.70:0/981935610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 msgr2=0x7f4940071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f4940071d10 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f49401600c0 tx=0x7f493002b880 comp rx=0 tx=0).stop 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 -- 172.21.15.70:0/981935610 shutdown_connections 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 --2- 172.21.15.70:0/981935610 >> v2:172.21.15.96:3300/0 conn(0x7f49400d0690 0x7f49400d0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400af3e0 0x7f49400af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 --2- 172.21.15.70:0/981935610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f4940071d10 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 -- 172.21.15.70:0/981935610 >> 172.21.15.70:0/981935610 conn(0x7f494006be00 msgr2=0x7f494006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 -- 172.21.15.70:0/981935610 shutdown_connections 2024-04-15T15:17:09.865 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.859+0000 7f4946a35640 1 -- 172.21.15.70:0/981935610 wait complete. 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.860+0000 7f4946a35640 1 Processor -- start 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.860+0000 7f4946a35640 1 -- start start 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49400af3e0 0x7f49400b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400d0690 0x7f494014c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f494015cd10 con 0x7f4940072d80 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f494015cb90 con 0x7f49400af3e0 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4946a35640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f494006e060 con 0x7f49400d0690 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45600/0 (socket says 172.21.15.70:45600) 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 -- 172.21.15.70:0/2460924570 learned_addr learned my addr 172.21.15.70:0/2460924570 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4944fd2640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400d0690 0x7f494014c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.866 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f4937fff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49400af3e0 0x7f49400b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 -- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400d0690 msgr2=0x7f494014c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400d0690 0x7f494014c210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 -- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49400af3e0 msgr2=0x7f49400b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49400af3e0 0x7f49400b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.861+0000 7f493ffff640 1 -- 172.21.15.70:0/2460924570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494014e1d0 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f493ffff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f493002b8c0 tx=0x7f49300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930032880 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4930032a20 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494014dbe0 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4930032e10 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.862+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4940157150 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.863+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49400689e0 con 0x7f4940072d80 2024-04-15T15:17:09.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.863+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f493004c060 con 0x7f4940072d80 2024-04-15T15:17:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.864+0000 7f493d7fa640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 0x7f492006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.864+0000 7f4937fff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 0x7f492006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.864+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f49300adae0 con 0x7f4940072d80 2024-04-15T15:17:09.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.864+0000 7f4937fff640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 0x7f492006ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f494009ce20 tx=0x7f4928008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:09.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:09.868+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f494014e1d0 con 0x7f4940072d80 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.044+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4940072380 con 0x7f4940072d80 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.044+0000 7f493d7fa640 1 -- 172.21.15.70:0/2460924570 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7f4930040070 con 0x7f4940072d80 2024-04-15T15:17:10.046 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 msgr2=0x7f492006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 0x7f492006ee40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f494009ce20 tx=0x7f4928008040 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 msgr2=0x7f494014d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f493002b8c0 tx=0x7f49300046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 shutdown_connections 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f49400d0690 0x7f494014c210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f49400af3e0 0x7f49400b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f492006c980 0x7f492006ee40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 --2- 172.21.15.70:0/2460924570 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4940072d80 0x7f494014d000 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 >> 172.21.15.70:0/2460924570 conn(0x7f494006be00 msgr2=0x7f494015dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 shutdown_connections 2024-04-15T15:17:10.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.046+0000 7f4946a35640 1 -- 172.21.15.70:0/2460924570 wait complete. 2024-04-15T15:17:10.060 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:10.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.164+0000 7f959d7aa640 1 Processor -- start 2024-04-15T15:17:10.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.164+0000 7f959d7aa640 1 -- start start 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f959d7aa640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f95980d2360 0x7f95980d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f959d7aa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f959815eae0 0x7f95980d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f959d7aa640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95980b47a0 con 0x7f959815e700 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f959d7aa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f95980b53b0 con 0x7f959815eae0 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f959d7aa640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f95980b4a80 con 0x7f95980d2360 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f959815eae0 0x7f95980d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f959815eae0 0x7f95980d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45602/0 (socket says 172.21.15.70:45602) 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 -- 172.21.15.70:0/3633794717 learned_addr learned my addr 172.21.15.70:0/3633794717 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 -- 172.21.15.70:0/3633794717 >> v2:172.21.15.96:3300/0 conn(0x7f95980d2360 msgr2=0x7f95980d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 --2- 172.21.15.70:0/3633794717 >> v2:172.21.15.96:3300/0 conn(0x7f95980d2360 0x7f95980d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 -- 172.21.15.70:0/3633794717 >> v1:172.21.15.96:6789/0 conn(0x7f959815e700 legacy=0x7f95980b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.165+0000 7f95977fe640 1 -- 172.21.15.70:0/3633794717 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95980b4d60 con 0x7f959815eae0 2024-04-15T15:17:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f95977fe640 1 --2- 172.21.15.70:0/3633794717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f95980d30a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f959808c7b0 tx=0x7f959002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576a4e14aab0b156 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f95967fc640 1 -- 172.21.15.70:0/3633794717 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959002ca80 con 0x7f959815eae0 2024-04-15T15:17:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f95967fc640 1 -- 172.21.15.70:0/3633794717 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959002cc20 con 0x7f959815eae0 2024-04-15T15:17:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f95967fc640 1 -- 172.21.15.70:0/3633794717 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95900343f0 con 0x7f959815eae0 2024-04-15T15:17:10.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 -- 172.21.15.70:0/3633794717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 msgr2=0x7f95980d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 --2- 172.21.15.70:0/3633794717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f95980d30a0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f959808c7b0 tx=0x7f959002bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 -- 172.21.15.70:0/3633794717 shutdown_connections 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 --2- 172.21.15.70:0/3633794717 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f95980d30a0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 --2- 172.21.15.70:0/3633794717 >> v2:172.21.15.96:3300/0 conn(0x7f95980d2360 0x7f95980d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 -- 172.21.15.70:0/3633794717 >> 172.21.15.70:0/3633794717 conn(0x7f959808bf60 msgr2=0x7f95980d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 -- 172.21.15.70:0/3633794717 shutdown_connections 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.166+0000 7f959d7aa640 1 -- 172.21.15.70:0/3633794717 wait complete. 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 Processor -- start 2024-04-15T15:17:10.170 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 -- start start 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95980d2360 0x7f95980cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f959815e700 0x7f95980ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f95980d1d40 con 0x7f959815eae0 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f95980b4450 con 0x7f95980d2360 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.168+0000 7f959d7aa640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f95980d1ec0 con 0x7f959815e700 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45604/0 (socket says 172.21.15.70:45604) 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 -- 172.21.15.70:0/1899560973 learned_addr learned my addr 172.21.15.70:0/1899560973 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:10.172 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 -- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f959815e700 msgr2=0x7f95980ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f959815e700 0x7f95980ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 -- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95980d2360 msgr2=0x7f95980cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95980d2360 0x7f95980cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 -- 172.21.15.70:0/1899560973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95980cb6f0 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f95977fe640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9590036040 tx=0x7f9590004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590043020 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959003e450 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9590009510 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95980cb9d0 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.169+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95980cbf90 con 0x7f959815eae0 2024-04-15T15:17:10.173 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.170+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95980689e0 con 0x7f959815eae0 2024-04-15T15:17:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.171+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9590046020 con 0x7f959815eae0 2024-04-15T15:17:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.171+0000 7f958ffff640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 0x7f957806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.172+0000 7f9596ffd640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 0x7f957806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.172+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f95900ae740 con 0x7f959815eae0 2024-04-15T15:17:10.174 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.172+0000 7f9596ffd640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 0x7f957806ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95880015b0 tx=0x7f95880079e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.177 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.176+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f959004d030 con 0x7f959815eae0 2024-04-15T15:17:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.367+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f95980735c0 con 0x7f959815eae0 2024-04-15T15:17:10.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.367+0000 7f958ffff640 1 -- 172.21.15.70:0/1899560973 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7f959007c980 con 0x7f959815eae0 2024-04-15T15:17:10.369 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6835/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:10.370 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6833/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:10.370 INFO:teuthology.orchestra.run.smithi070.stderr:listed 2 entries 2024-04-15T15:17:10.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 msgr2=0x7f957806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 0x7f957806ee40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f95880015b0 tx=0x7f95880079e0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 msgr2=0x7f9598152ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f9590036040 tx=0x7f9590004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 shutdown_connections 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f957806c980 0x7f957806ee40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f959815eae0 0x7f9598152ae0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f959815e700 0x7f95980ca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 --2- 172.21.15.70:0/1899560973 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f95980d2360 0x7f95980cdba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.369+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 >> 172.21.15.70:0/1899560973 conn(0x7f959808bf60 msgr2=0x7f95980b68e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.370+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 shutdown_connections 2024-04-15T15:17:10.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.370+0000 7f959d7aa640 1 -- 172.21.15.70:0/1899560973 wait complete. 2024-04-15T15:17:10.382 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi070.front.sepia.ceph.com... 2024-04-15T15:17:10.383 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:10.383 DEBUG:teuthology.orchestra.run.smithi070:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-15T15:17:10.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi070.stderr:ceph-fuse[55717]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:10.411 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:10.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:10.412 DEBUG:teuthology.orchestra.run.smithi070:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-15T15:17:10.582 INFO:tasks.cephfs.mount:Terminating background process 2024-04-15T15:17:10.582 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:17:10.582 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:10.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.725+0000 7efe99fb5640 1 Processor -- start 2024-04-15T15:17:10.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.725+0000 7efe99fb5640 1 -- start start 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe99fb5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efe94158770 0x7efe9415bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe99fb5640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7efe94072e30 0x7efe9406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe99fb5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe9415c1d0 con 0x7efe940b5230 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe99fb5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7efe940b4da0 con 0x7efe94158770 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe99fb5640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7efe940b4190 con 0x7efe94072e30 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe937fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7efe94158770 0x7efe9415bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe8bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efe940b5230 0x7efe94072720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60442/0 (socket says 172.21.15.70:60442) 2024-04-15T15:17:10.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe8bfff640 1 -- 172.21.15.70:0/951600422 learned_addr learned my addr 172.21.15.70:0/951600422 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe93fff640 1 --2- 172.21.15.70:0/951600422 >> v2:172.21.15.96:3300/0 conn(0x7efe94072e30 0x7efe9406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe937fe640 1 -- 172.21.15.70:0/951600422 >> v2:172.21.15.96:3300/0 conn(0x7efe94072e30 msgr2=0x7efe9406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe937fe640 1 --2- 172.21.15.70:0/951600422 >> v2:172.21.15.96:3300/0 conn(0x7efe94072e30 0x7efe9406dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe937fe640 1 -- 172.21.15.70:0/951600422 >> v1:172.21.15.70:6789/0 conn(0x7efe940b5230 legacy=0x7efe94072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe937fe640 1 -- 172.21.15.70:0/951600422 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe940b4470 con 0x7efe94158770 2024-04-15T15:17:10.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.726+0000 7efe92ffd640 1 -- 172.21.15.70:0/951600422 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1611387492 0 0) 0x7efe7c001110 con 0x7efe940b5230 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.727+0000 7efe937fe640 1 --2- 172.21.15.70:0/951600422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe94158770 0x7efe9415bc90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7efe94072640 tx=0x7efe8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48673718bf08c772 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.727+0000 7efe92ffd640 1 -- 172.21.15.70:0/951600422 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84035020 con 0x7efe94158770 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.727+0000 7efe92ffd640 1 -- 172.21.15.70:0/951600422 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe84031830 con 0x7efe94158770 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.727+0000 7efe92ffd640 1 -- 172.21.15.70:0/951600422 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe84031c00 con 0x7efe94158770 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 -- 172.21.15.70:0/951600422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe94158770 msgr2=0x7efe9415bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 --2- 172.21.15.70:0/951600422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe94158770 0x7efe9415bc90 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7efe94072640 tx=0x7efe8402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 -- 172.21.15.70:0/951600422 shutdown_connections 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 --2- 172.21.15.70:0/951600422 >> v2:172.21.15.96:3300/0 conn(0x7efe94072e30 0x7efe9406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 --2- 172.21.15.70:0/951600422 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe94158770 0x7efe9415bc90 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 -- 172.21.15.70:0/951600422 >> 172.21.15.70:0/951600422 conn(0x7efe9406be00 msgr2=0x7efe9406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 -- 172.21.15.70:0/951600422 shutdown_connections 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.728+0000 7efe99fb5640 1 -- 172.21.15.70:0/951600422 wait complete. 2024-04-15T15:17:10.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 Processor -- start 2024-04-15T15:17:10.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 -- start start 2024-04-15T15:17:10.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe94072e30 0x7efe9414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efe94158770 0x7efe9415ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe94071fa0 con 0x7efe940b5230 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efe940b4a30 con 0x7efe94072e30 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.729+0000 7efe99fb5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efe94158b50 con 0x7efe94158770 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45608/0 (socket says 172.21.15.70:45608) 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 -- 172.21.15.70:0/4102093981 learned_addr learned my addr 172.21.15.70:0/4102093981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe937fe640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe94072e30 0x7efe9414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe93fff640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efe94158770 0x7efe9415ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 -- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efe94158770 msgr2=0x7efe9415ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efe94158770 0x7efe9415ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.735 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 -- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe94072e30 msgr2=0x7efe9414d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe94072e30 0x7efe9414d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 -- 172.21.15.70:0/4102093981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe9414d840 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe8bfff640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c780 tx=0x7efe7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c018070 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe7c0072d0 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe7c0047e0 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe940cd280 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.730+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe940cd840 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.731+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe940689e0 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.732+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7efe7c004980 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.732+0000 7efe90ff9640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 0x7efe7406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.732+0000 7efe937fe640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 0x7efe7406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.732+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7efe7c08af20 con 0x7efe940b5230 2024-04-15T15:17:10.736 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.733+0000 7efe937fe640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 0x7efe7406ee70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efe84036000 tx=0x7efe8403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:10.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.737+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efe9414d840 con 0x7efe940b5230 2024-04-15T15:17:10.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.914+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efe940735d0 con 0x7efe940b5230 2024-04-15T15:17:10.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.915+0000 7efe90ff9640 1 -- 172.21.15.70:0/4102093981 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7efe9415c1d0 con 0x7efe940b5230 2024-04-15T15:17:10.917 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:10.917 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:10.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 msgr2=0x7efe7406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 0x7efe7406ee70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efe84036000 tx=0x7efe8403c040 comp rx=0 tx=0).stop 2024-04-15T15:17:10.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 msgr2=0x7efe9415e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7efe7c00c780 tx=0x7efe7c00cc40 comp rx=0 tx=0).stop 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 shutdown_connections 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efe94158770 0x7efe9415ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efe7406c9b0 0x7efe7406ee70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe940b5230 0x7efe9415e790 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 --2- 172.21.15.70:0/4102093981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efe94072e30 0x7efe9414d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 >> 172.21.15.70:0/4102093981 conn(0x7efe9406be00 msgr2=0x7efe94157620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 shutdown_connections 2024-04-15T15:17:10.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:10.917+0000 7efe99fb5640 1 -- 172.21.15.70:0/4102093981 wait complete. 2024-04-15T15:17:10.931 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:10.931 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:11.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.036+0000 7efdad0ba640 1 Processor -- start 2024-04-15T15:17:11.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.036+0000 7efdad0ba640 1 -- start start 2024-04-15T15:17:11.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.036+0000 7efdad0ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efda8160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efdad0ba640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efda8070720 con 0x7efda80701f0 2024-04-15T15:17:11.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efdad0ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efda8070a90 con 0x7efda815fb30 2024-04-15T15:17:11.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efdad0ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efda809ba90 con 0x7efda809a9d0 2024-04-15T15:17:11.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efd9e59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7efda80701f0 0x7efda809adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:60446/0 (socket says 172.21.15.70:60446) 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efd9e59c640 1 -- 172.21.15.70:0/1865978155 learned_addr learned my addr 172.21.15.70:0/1865978155 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3601075517 0 0) 0x7efda8070720 con 0x7efda80701f0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd94003520 con 0x7efda80701f0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efda8160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1416124515 0 0) 0x7efd94003520 con 0x7efda80701f0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 msgr2=0x7efda8160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efda8160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 >> v1:172.21.15.96:6789/0 conn(0x7efda815fb30 legacy=0x7efda80d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda8070d70 con 0x7efda80701f0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7efd90001670 con 0x7efda80701f0 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efda8160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 >> v1:172.21.15.70:6789/0 conn(0x7efda80701f0 legacy=0x7efda809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efd94003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efd94004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd94004d40 0x7efd940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efd9400bad0 con 0x7efd940039e0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efd9400bc80 con 0x7efd94004d40 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efd9400be80 con 0x7efda809a9d0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efd94003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efd9e59c640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efd94004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.037+0000 7efda759e640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd94004d40 0x7efd940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 -- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 msgr2=0x7efd94004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda809a9d0 0x7efd94004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 -- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd94004d40 msgr2=0x7efd940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efd94004d40 0x7efd940051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 -- 172.21.15.70:0/1865978155 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd94003520 con 0x7efd940039e0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efd94003e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efd98006fd0 tx=0x7efd94005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e33662e10255733d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.038+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9800f040 con 0x7efd940039e0 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9802e070 con 0x7efd940039e0 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efda659c640 1 -- 172.21.15.70:0/1865978155 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98004450 con 0x7efd940039e0 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 -- 172.21.15.70:0/1865978155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 msgr2=0x7efd94003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1865978155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efd94003e00 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7efd98006fd0 tx=0x7efd94005830 comp rx=0 tx=0).stop 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efda759e640 1 -- 172.21.15.70:0/1865978155 reap_dead start 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 -- 172.21.15.70:0/1865978155 shutdown_connections 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 -- 172.21.15.70:0/1865978155 >> 172.21.15.70:0/1865978155 conn(0x7efda80d2460 msgr2=0x7efda80d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 -- 172.21.15.70:0/1865978155 shutdown_connections 2024-04-15T15:17:11.044 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.039+0000 7efdad0ba640 1 -- 172.21.15.70:0/1865978155 wait complete. 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.040+0000 7efdad0ba640 1 Processor -- start 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 -- start start 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda80c7e00 0x7efda80c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda8161420 0x7efda81618a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efda815ffc0 con 0x7efd940039e0 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7efda80d08c0 con 0x7efda8161420 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efdad0ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7efda80b1c60 con 0x7efda80c7e00 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efda6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efda6d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45628/0 (socket says 172.21.15.70:45628) 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.041+0000 7efda6d9d640 1 -- 172.21.15.70:0/1087610290 learned_addr learned my addr 172.21.15.70:0/1087610290 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda759e640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda8161420 0x7efda81618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efd9e59c640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda80c7e00 0x7efda80c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 -- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda80c7e00 msgr2=0x7efda80c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda80c7e00 0x7efda80c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 -- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda8161420 msgr2=0x7efda81618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda8161420 0x7efda81618a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 -- 172.21.15.70:0/1087610290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda80c8880 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efda6d9d640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7efd980098d0 tx=0x7efd98004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98032680 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd98032820 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98032c10 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda80ccb30 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.042+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda80cd0f0 con 0x7efd940039e0 2024-04-15T15:17:11.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.043+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda8161de0 con 0x7efd940039e0 2024-04-15T15:17:11.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.044+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7efd9803e430 con 0x7efd940039e0 2024-04-15T15:17:11.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.044+0000 7efd9ffff640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 0x7efd8806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.044+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7efd980ae5c0 con 0x7efd940039e0 2024-04-15T15:17:11.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.044+0000 7efd9e59c640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 0x7efd8806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.044+0000 7efd9e59c640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 0x7efd8806ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efd90000c00 tx=0x7efd90000f80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.050 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.048+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7efd98042030 con 0x7efd940039e0 2024-04-15T15:17:11.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.222+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efda80689e0 con 0x7efd940039e0 2024-04-15T15:17:11.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.222+0000 7efd9ffff640 1 -- 172.21.15.70:0/1087610290 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7efd9800f000 con 0x7efd940039e0 2024-04-15T15:17:11.224 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:17:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 msgr2=0x7efd8806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 0x7efd8806ee70 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efd90000c00 tx=0x7efd90000f80 comp rx=0 tx=0).stop 2024-04-15T15:17:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 msgr2=0x7efda815cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7efd980098d0 tx=0x7efd98004730 comp rx=0 tx=0).stop 2024-04-15T15:17:11.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 shutdown_connections 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7efda8161420 0x7efda81618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7efda80c7e00 0x7efda80c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7efd8806c9b0 0x7efd8806ee70 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 --2- 172.21.15.70:0/1087610290 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efd940039e0 0x7efda815cc30 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 >> 172.21.15.70:0/1087610290 conn(0x7efda80d2460 msgr2=0x7efda80b1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 shutdown_connections 2024-04-15T15:17:11.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.224+0000 7efdad0ba640 1 -- 172.21.15.70:0/1087610290 wait complete. 2024-04-15T15:17:11.237 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.341+0000 7fed8f6e4640 1 Processor -- start 2024-04-15T15:17:11.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 -- start start 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed880d2360 0x7fed880d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed880d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed880b47a0 con 0x7fed8815e700 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fed880b53b0 con 0x7fed880d2360 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.342+0000 7fed8f6e4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fed880b4a80 con 0x7fed8815eae0 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed880d2360 0x7fed880d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fed880d2360 0x7fed880d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45630/0 (socket says 172.21.15.70:45630) 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 -- 172.21.15.70:0/3208654026 learned_addr learned my addr 172.21.15.70:0/3208654026 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 -- 172.21.15.70:0/3208654026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 msgr2=0x7fed880d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8dc81640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed880d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed880d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 -- 172.21.15.70:0/3208654026 >> v1:172.21.15.96:6789/0 conn(0x7fed8815e700 legacy=0x7fed880b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.343+0000 7fed8cc7f640 1 -- 172.21.15.70:0/3208654026 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed880b4d60 con 0x7fed880d2360 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8cc7f640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed880d2360 0x7fed880d2b60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fed8808c7b0 tx=0x7fed7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd86c27ede7e9905 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed87fff640 1 -- 172.21.15.70:0/3208654026 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035020 con 0x7fed880d2360 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed87fff640 1 -- 172.21.15.70:0/3208654026 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7c031640 con 0x7fed880d2360 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed87fff640 1 -- 172.21.15.70:0/3208654026 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c02d040 con 0x7fed880d2360 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 -- 172.21.15.70:0/3208654026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed880d2360 msgr2=0x7fed880d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed880d2360 0x7fed880d2b60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fed8808c7b0 tx=0x7fed7c02b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 -- 172.21.15.70:0/3208654026 shutdown_connections 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed880d30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/3208654026 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed880d2360 0x7fed880d2b60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 -- 172.21.15.70:0/3208654026 >> 172.21.15.70:0/3208654026 conn(0x7fed8808bf60 msgr2=0x7fed880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 -- 172.21.15.70:0/3208654026 shutdown_connections 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.344+0000 7fed8f6e4640 1 -- 172.21.15.70:0/3208654026 wait complete. 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 Processor -- start 2024-04-15T15:17:11.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 -- start start 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed880d2360 0x7fed8815aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed88160d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fed880d1d40 con 0x7fed8815e700 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fed880b4450 con 0x7fed880d2360 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8f6e4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fed880d1ec0 con 0x7fed8815eae0 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8cc7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8cc7f640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45638/0 (socket says 172.21.15.70:45638) 2024-04-15T15:17:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.346+0000 7fed8cc7f640 1 -- 172.21.15.70:0/2374471025 learned_addr learned my addr 172.21.15.70:0/2374471025 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8dc81640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed88160d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 -- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 msgr2=0x7fed88160d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed88160d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 -- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed880d2360 msgr2=0x7fed8815aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8d480640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed880d2360 0x7fed8815aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed880d2360 0x7fed8815aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 -- 172.21.15.70:0/2374471025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed880b7800 con 0x7fed8815e700 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8cc7f640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fed7c02b930 tx=0x7fed7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c040070 con 0x7fed8815e700 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed7c039c30 con 0x7fed8815e700 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed880b7ae0 con 0x7fed8815e700 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed7c035040 con 0x7fed8815e700 2024-04-15T15:17:11.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.347+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed880b80a0 con 0x7fed8815e700 2024-04-15T15:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.348+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed880689e0 con 0x7fed8815e700 2024-04-15T15:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.349+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fed7c038020 con 0x7fed8815e700 2024-04-15T15:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.349+0000 7fed85ffb640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 0x7fed7406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.349+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fed7c0ad970 con 0x7fed8815e700 2024-04-15T15:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.349+0000 7fed8d480640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 0x7fed7406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.350+0000 7fed8d480640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 0x7fed7406ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fed78007fd0 tx=0x7fed78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.354+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fed7c0afe20 con 0x7fed8815e700 2024-04-15T15:17:11.529 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.527+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fed880ba500 con 0x7fed8815e700 2024-04-15T15:17:11.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.528+0000 7fed85ffb640 1 -- 172.21.15.70:0/2374471025 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7fed7c031610 con 0x7fed8815e700 2024-04-15T15:17:11.530 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:17:11.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 msgr2=0x7fed7406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 0x7fed7406ee40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fed78007fd0 tx=0x7fed78009040 comp rx=0 tx=0).stop 2024-04-15T15:17:11.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 msgr2=0x7fed880cd870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fed7c02b930 tx=0x7fed7c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 shutdown_connections 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fed8815eae0 0x7fed88160d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fed7406c980 0x7fed7406ee40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fed8815e700 0x7fed880cd870 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 --2- 172.21.15.70:0/2374471025 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fed880d2360 0x7fed8815aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 >> 172.21.15.70:0/2374471025 conn(0x7fed8808bf60 msgr2=0x7fed880b10f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 shutdown_connections 2024-04-15T15:17:11.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.530+0000 7fed8f6e4640 1 -- 172.21.15.70:0/2374471025 wait complete. 2024-04-15T15:17:11.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:11.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.646+0000 7fa69f6d3640 1 Processor -- start 2024-04-15T15:17:11.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 -- start start 2024-04-15T15:17:11.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6980d0690 0x7fa6980d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 0x7fa69815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6980d0fb0 con 0x7fa69815ef70 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa69815cb50 con 0x7fa6980d0690 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.647+0000 7fa69f6d3640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa69815bfd0 con 0x7fa6980b3160 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6980d0690 0x7fa6980d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa6980d0690 0x7fa6980d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45640/0 (socket says 172.21.15.70:45640) 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 -- 172.21.15.70:0/278807068 learned_addr learned my addr 172.21.15.70:0/278807068 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69cc6e640 1 --2- 172.21.15.70:0/278807068 >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 0x7fa69815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 -- 172.21.15.70:0/278807068 >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 msgr2=0x7fa69815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 --2- 172.21.15.70:0/278807068 >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 0x7fa69815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 -- 172.21.15.70:0/278807068 >> v1:172.21.15.96:6789/0 conn(0x7fa69815ef70 legacy=0x7fa69815f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69d46f640 1 -- 172.21.15.70:0/278807068 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69815c2b0 con 0x7fa6980d0690 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.648+0000 7fa69cc6e640 1 --2- 172.21.15.70:0/278807068 >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 0x7fa69815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69d46f640 1 --2- 172.21.15.70:0/278807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa6980d0a70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa69806fec0 tx=0x7fa69002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3dcccf81ab98115 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.651 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa697fff640 1 -- 172.21.15.70:0/278807068 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690034070 con 0x7fa6980d0690 2024-04-15T15:17:11.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa697fff640 1 -- 172.21.15.70:0/278807068 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa690030860 con 0x7fa6980d0690 2024-04-15T15:17:11.652 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa697fff640 1 -- 172.21.15.70:0/278807068 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690030c30 con 0x7fa6980d0690 2024-04-15T15:17:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 -- 172.21.15.70:0/278807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 msgr2=0x7fa6980d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/278807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa6980d0a70 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa69806fec0 tx=0x7fa69002b9a0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 -- 172.21.15.70:0/278807068 shutdown_connections 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/278807068 >> v2:172.21.15.96:3300/0 conn(0x7fa6980b3160 0x7fa69815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/278807068 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa6980d0a70 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 -- 172.21.15.70:0/278807068 >> 172.21.15.70:0/278807068 conn(0x7fa69806be00 msgr2=0x7fa69806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 -- 172.21.15.70:0/278807068 shutdown_connections 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.649+0000 7fa69f6d3640 1 -- 172.21.15.70:0/278807068 wait complete. 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.650+0000 7fa69f6d3640 1 Processor -- start 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 -- start start 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6980b3160 0x7fa6980cf610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa69815ef70 0x7fa69815f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa6980b4fb0 con 0x7fa6980d0690 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa69815c870 con 0x7fa6980b3160 2024-04-15T15:17:11.657 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69f6d3640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa69815bbd0 con 0x7fa69815ef70 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69cc6e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69cc6e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45650/0 (socket says 172.21.15.70:45650) 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.651+0000 7fa69cc6e640 1 -- 172.21.15.70:0/897340872 learned_addr learned my addr 172.21.15.70:0/897340872 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69dc70640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa69815ef70 0x7fa69815f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 -- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa69815ef70 msgr2=0x7fa69815f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa69815ef70 0x7fa69815f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 -- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6980b3160 msgr2=0x7fa6980cf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6980b3160 0x7fa6980cf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 -- 172.21.15.70:0/897340872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69815ce70 con 0x7fa6980d0690 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69cc6e640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00d800 tx=0x7fa68c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c004030 con 0x7fa6980d0690 2024-04-15T15:17:11.658 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c0041d0 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69815d150 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c0096a0 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.652+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa69815da40 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.653+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6980689e0 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.654+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa68c009840 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.654+0000 7fa695ffb640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 0x7fa68406ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.654+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa68c08c060 con 0x7fa6980d0690 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.654+0000 7fa69d46f640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 0x7fa68406ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.659 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.655+0000 7fa69d46f640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 0x7fa68406ee70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa690036000 tx=0x7fa690030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:11.660 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.659+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa68c05a2a0 con 0x7fa6980d0690 2024-04-15T15:17:11.768 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:11.771 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.837+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fa6980d0d90 con 0x7fa6980d0690 2024-04-15T15:17:11.839 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.838+0000 7fa695ffb640 1 -- 172.21.15.70:0/897340872 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fa68c059ba0 con 0x7fa6980d0690 2024-04-15T15:17:11.840 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 msgr2=0x7fa68406ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 0x7fa68406ee70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa690036000 tx=0x7fa690030000 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 msgr2=0x7fa698146e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa68c00d800 tx=0x7fa68c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 shutdown_connections 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa69815ef70 0x7fa69815f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa68406c9b0 0x7fa68406ee70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa6980d0690 0x7fa698146e90 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 --2- 172.21.15.70:0/897340872 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa6980b3160 0x7fa6980cf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 >> 172.21.15.70:0/897340872 conn(0x7fa69806be00 msgr2=0x7fa6980ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:11.842 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 shutdown_connections 2024-04-15T15:17:11.843 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.840+0000 7fa69f6d3640 1 -- 172.21.15.70:0/897340872 wait complete. 2024-04-15T15:17:11.853 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-15T15:17:11.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.994+0000 7fd78b4e4640 1 Processor -- start 2024-04-15T15:17:11.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 -- start start 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7840d0fb0 con 0x7fd7840b3160 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd78415bda0 con 0x7fd7840ae280 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.995+0000 7fd78b4e4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd78415cb50 con 0x7fd7840d0690 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789a81640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789280640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789a81640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:48990/0 (socket says 172.21.15.70:48990) 2024-04-15T15:17:11.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789a81640 1 -- 172.21.15.70:0/251884439 learned_addr learned my addr 172.21.15.70:0/251884439 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd788a7f640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fd7840b3160 0x7fd78415b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:45988/0 (socket says 172.21.15.70:45988) 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789280640 1 -- 172.21.15.70:0/251884439 >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 msgr2=0x7fd7840ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789280640 1 --2- 172.21.15.70:0/251884439 >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789280640 1 -- 172.21.15.70:0/251884439 >> v1:172.21.15.96:6789/0 conn(0x7fd7840b3160 legacy=0x7fd78415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789280640 1 -- 172.21.15.70:0/251884439 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78415bfd0 con 0x7fd7840d0690 2024-04-15T15:17:11.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.996+0000 7fd789a81640 1 --2- 172.21.15.70:0/251884439 >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:12.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.997+0000 7fd789280640 1 --2- 172.21.15.70:0/251884439 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840d0a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd7800097a0 tx=0x7fd78002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37523c0d1280becd server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.997+0000 7fd777fff640 1 -- 172.21.15.70:0/251884439 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780035020 con 0x7fd7840d0690 2024-04-15T15:17:12.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.997+0000 7fd777fff640 1 -- 172.21.15.70:0/251884439 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd780031830 con 0x7fd7840d0690 2024-04-15T15:17:12.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.997+0000 7fd777fff640 1 -- 172.21.15.70:0/251884439 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780031c00 con 0x7fd7840d0690 2024-04-15T15:17:12.000 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 -- 172.21.15.70:0/251884439 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 msgr2=0x7fd7840d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/251884439 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840d0a70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd7800097a0 tx=0x7fd78002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 -- 172.21.15.70:0/251884439 shutdown_connections 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/251884439 >> v2:172.21.15.96:3300/0 conn(0x7fd7840ae280 0x7fd7840ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/251884439 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840d0a70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 -- 172.21.15.70:0/251884439 >> 172.21.15.70:0/251884439 conn(0x7fd78406be00 msgr2=0x7fd78406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 -- 172.21.15.70:0/251884439 shutdown_connections 2024-04-15T15:17:12.001 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.998+0000 7fd78b4e4640 1 -- 172.21.15.70:0/251884439 wait complete. 2024-04-15T15:17:12.003 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.999+0000 7fd78b4e4640 1 Processor -- start 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:11.999+0000 7fd78b4e4640 1 -- start start 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7840b3160 0x7fd784156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd78415c870 con 0x7fd7840ae280 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd78415c590 con 0x7fd7840b3160 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd78b4e4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd7840b4fb0 con 0x7fd7840d0690 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:45652/0 (socket says 172.21.15.70:45652) 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 -- 172.21.15.70:0/105887843 learned_addr learned my addr 172.21.15.70:0/105887843 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789a81640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd788a7f640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7840b3160 0x7fd784156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.004 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 -- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 msgr2=0x7fd7840ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 -- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7840b3160 msgr2=0x7fd784156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7840b3160 0x7fd784156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.000+0000 7fd789280640 1 -- 172.21.15.70:0/105887843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7841556d0 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd789280640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd780000c00 tx=0x7fd7800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780040070 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd78003a850 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd784149790 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd780035040 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.001+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd784155f30 con 0x7fd7840ae280 2024-04-15T15:17:12.005 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.002+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7840689e0 con 0x7fd7840ae280 2024-04-15T15:17:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.002+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd78004b060 con 0x7fd7840ae280 2024-04-15T15:17:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.003+0000 7fd775ffb640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 0x7fd76806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.003+0000 7fd788a7f640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 0x7fd76806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.003+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fd7800ad550 con 0x7fd7840ae280 2024-04-15T15:17:12.006 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.003+0000 7fd788a7f640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 0x7fd76806ee70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd784070740 tx=0x7fd778007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.009 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.007+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd780031800 con 0x7fd7840ae280 2024-04-15T15:17:12.198 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.196+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd7840739e0 con 0x7fd7840ae280 2024-04-15T15:17:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.197+0000 7fd775ffb640 1 -- 172.21.15.70:0/105887843 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v42) ==== 80+0+122 (secure 0 0 0) 0x7fd78007b1d0 con 0x7fd7840ae280 2024-04-15T15:17:12.199 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6835/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:12.199 INFO:teuthology.orchestra.run.smithi070.stdout:172.21.15.96:6833/2334246723 2024-04-16T15:17:00.083643+0000 2024-04-15T15:17:12.199 INFO:teuthology.orchestra.run.smithi070.stderr:listed 2 entries 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 msgr2=0x7fd76806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 0x7fd76806ee70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd784070740 tx=0x7fd778007c80 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 msgr2=0x7fd784149250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fd780000c00 tx=0x7fd7800046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 shutdown_connections 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd7840d0690 0x7fd7840ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd7840b3160 0x7fd784156730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd76806c9b0 0x7fd76806ee70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 --2- 172.21.15.70:0/105887843 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd7840ae280 0x7fd784149250 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 >> 172.21.15.70:0/105887843 conn(0x7fd78406be00 msgr2=0x7fd78406dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.201 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 shutdown_connections 2024-04-15T15:17:12.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.199+0000 7fd78b4e4640 1 -- 172.21.15.70:0/105887843 wait complete. 2024-04-15T15:17:12.211 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi096.front.sepia.ceph.com... 2024-04-15T15:17:12.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.212 DEBUG:teuthology.orchestra.run.smithi096:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-15T15:17:12.244 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi096.stderr:ceph-fuse[45996]: fuse finished with error 0 and tester_r 0 2024-04-15T15:17:12.246 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:12.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:12.246 DEBUG:teuthology.orchestra.run.smithi096:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-15T15:17:12.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.522+0000 7f4141fe6640 1 Processor -- start 2024-04-15T15:17:12.524 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.522+0000 7f4141fe6640 1 -- start start 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f413c0701f0 0x7f413c15fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f413c09adb0 0x7f413c0d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f413c09ba90 con 0x7f413c0701f0 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f413c160070 con 0x7f413c09adb0 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f4141fe6640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f413c0d08c0 con 0x7f413c09a9d0 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f413c0701f0 0x7f413c15fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f413c0701f0 0x7f413c15fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48900/0 (socket says 172.21.15.70:48900) 2024-04-15T15:17:12.526 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 -- 172.21.15.70:0/142626154 learned_addr learned my addr 172.21.15.70:0/142626154 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413b7fe640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413bfff640 1 --2- 172.21.15.70:0/142626154 >> v2:172.21.15.96:3300/0 conn(0x7f413c09adb0 0x7f413c0d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 -- 172.21.15.70:0/142626154 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 msgr2=0x7f413c160420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c160420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 -- 172.21.15.70:0/142626154 >> v2:172.21.15.96:3300/0 conn(0x7f413c09adb0 msgr2=0x7f413c0d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 --2- 172.21.15.70:0/142626154 >> v2:172.21.15.96:3300/0 conn(0x7f413c09adb0 0x7f413c0d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.523+0000 7f413affd640 1 -- 172.21.15.70:0/142626154 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c070720 con 0x7f413c0701f0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.524+0000 7f413affd640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c0701f0 0x7f413c15fb30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4130009870 tx=0x7f413002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b5a94b7225efd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.524+0000 7f413a7fc640 1 -- 172.21.15.70:0/142626154 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4130035020 con 0x7f413c0701f0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.524+0000 7f413a7fc640 1 -- 172.21.15.70:0/142626154 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4130031640 con 0x7f413c0701f0 2024-04-15T15:17:12.527 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.524+0000 7f413a7fc640 1 -- 172.21.15.70:0/142626154 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413002d040 con 0x7f413c0701f0 2024-04-15T15:17:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 -- 172.21.15.70:0/142626154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c0701f0 msgr2=0x7f413c15fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c0701f0 0x7f413c15fb30 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f4130009870 tx=0x7f413002b8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 -- 172.21.15.70:0/142626154 shutdown_connections 2024-04-15T15:17:12.530 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 --2- 172.21.15.70:0/142626154 >> v2:172.21.15.96:3300/0 conn(0x7f413c09adb0 0x7f413c0d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c0701f0 0x7f413c15fb30 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 --2- 172.21.15.70:0/142626154 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c160420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 -- 172.21.15.70:0/142626154 >> 172.21.15.70:0/142626154 conn(0x7f413c0d2460 msgr2=0x7f413c0d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 -- 172.21.15.70:0/142626154 shutdown_connections 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.525+0000 7f4141fe6640 1 -- 172.21.15.70:0/142626154 wait complete. 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.526+0000 7f4141fe6640 1 Processor -- start 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.526+0000 7f4141fe6640 1 -- start start 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f413c0701f0 0x7f413c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c0b5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f413c071050 con 0x7f413c09adb0 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f413c070d70 con 0x7f413c0701f0 2024-04-15T15:17:12.531 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4141fe6640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f413c0b1c60 con 0x7f413c09a9d0 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48906/0 (socket says 172.21.15.70:48906) 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 -- 172.21.15.70:0/3256734912 learned_addr learned my addr 172.21.15.70:0/3256734912 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413b7fe640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f413c0701f0 0x7f413c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413affd640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c0b5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 -- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 msgr2=0x7f413c0b5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c0b5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 -- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f413c0701f0 msgr2=0x7f413c15b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.532 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f413c0701f0 0x7f413c15b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 -- 172.21.15.70:0/3256734912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c15bd50 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f413bfff640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4134007e90 tx=0x7f413400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134014420 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.527+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4134019070 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.528+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c15db90 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.528+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134014800 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.528+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c15e150 con 0x7f413c09adb0 2024-04-15T15:17:12.533 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.528+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c0689e0 con 0x7f413c09adb0 2024-04-15T15:17:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.529+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f41340149a0 con 0x7f413c09adb0 2024-04-15T15:17:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.529+0000 7f4123fff640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 0x7f412806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.529+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f413408c3d0 con 0x7f413c09adb0 2024-04-15T15:17:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.529+0000 7f413b7fe640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 0x7f412806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.534 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.530+0000 7f413b7fe640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 0x7f412806ee40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f412c006fd0 tx=0x7f412c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.535 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.534+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f413405a050 con 0x7f413c09adb0 2024-04-15T15:17:12.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.747+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f413c0d3370 con 0x7f413c09adb0 2024-04-15T15:17:12.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.747+0000 7f4123fff640 1 -- 172.21.15.70:0/3256734912 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5179 (secure 0 0 0) 0x7f413405e350 con 0x7f413c09adb0 2024-04-15T15:17:12.749 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:12.749 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:03.637138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5518,"qdb_cluster":[5518]},"id":2}]} 2024-04-15T15:17:12.750 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 20 2024-04-15T15:17:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 msgr2=0x7f412806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 0x7f412806ee40 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f412c006fd0 tx=0x7f412c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 msgr2=0x7f413c0b5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f4134007e90 tx=0x7f413400e4b0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 shutdown_connections 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f412806c980 0x7f412806ee40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f413c09adb0 0x7f413c0b5a50 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f413c09a9d0 0x7f413c0b5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 --2- 172.21.15.70:0/3256734912 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f413c0701f0 0x7f413c15b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 >> 172.21.15.70:0/3256734912 conn(0x7f413c0d2460 msgr2=0x7f413c0d0c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 shutdown_connections 2024-04-15T15:17:12.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.749+0000 7f4141fe6640 1 -- 172.21.15.70:0/3256734912 wait complete. 2024-04-15T15:17:12.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.865+0000 7fc454c69640 1 Processor -- start 2024-04-15T15:17:12.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.865+0000 7fc454c69640 1 -- start start 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4500d0690 0x7fc4500d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 0x7fc45015b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 0x7fc45015f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fc4500d0fb0 con 0x7fc4500d0690 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc45015cb50 con 0x7fc4500b3160 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc454c69640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc45015bfd0 con 0x7fc45015ef70 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4500d0690 0x7fc4500d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fc4500d0690 0x7fc4500d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48910/0 (socket says 172.21.15.70:48910) 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 -- 172.21.15.70:0/811176544 learned_addr learned my addr 172.21.15.70:0/811176544 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:12.870 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc445d9b640 1 --2- 172.21.15.70:0/811176544 >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 0x7fc45015b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44ed9d640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 0x7fc45015f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 -- 172.21.15.70:0/811176544 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 msgr2=0x7fc45015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 0x7fc45015f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 -- 172.21.15.70:0/811176544 >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 msgr2=0x7fc45015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 --2- 172.21.15.70:0/811176544 >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 0x7fc45015b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44e59c640 1 -- 172.21.15.70:0/811176544 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45015c2b0 con 0x7fc4500d0690 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc445d9b640 1 --2- 172.21.15.70:0/811176544 >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 0x7fc45015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.866+0000 7fc44ed9d640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 0x7fc45015f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc44e59c640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500d0a70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc45006fec0 tx=0x7fc44002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd8c098d7374ba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc44dd9b640 1 -- 172.21.15.70:0/811176544 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440035020 con 0x7fc4500d0690 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc44dd9b640 1 -- 172.21.15.70:0/811176544 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc440031870 con 0x7fc4500d0690 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc44dd9b640 1 -- 172.21.15.70:0/811176544 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc440031c40 con 0x7fc4500d0690 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc454c69640 1 -- 172.21.15.70:0/811176544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 msgr2=0x7fc4500d0a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.871 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.867+0000 7fc454c69640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500d0a70 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fc45006fec0 tx=0x7fc44002b910 comp rx=0 tx=0).stop 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 -- 172.21.15.70:0/811176544 shutdown_connections 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc45015ef70 0x7fc45015f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 --2- 172.21.15.70:0/811176544 >> v2:172.21.15.96:3300/0 conn(0x7fc4500b3160 0x7fc45015b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 --2- 172.21.15.70:0/811176544 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500d0a70 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 -- 172.21.15.70:0/811176544 >> 172.21.15.70:0/811176544 conn(0x7fc45006be00 msgr2=0x7fc45006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 -- 172.21.15.70:0/811176544 shutdown_connections 2024-04-15T15:17:12.872 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.868+0000 7fc454c69640 1 -- 172.21.15.70:0/811176544 wait complete. 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.869+0000 7fc454c69640 1 Processor -- start 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.869+0000 7fc454c69640 1 -- start start 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4500b3160 0x7fc4501e5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc45015ef70 0x7fc450153860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4500b4fb0 con 0x7fc4500d0690 2024-04-15T15:17:12.875 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc45015c870 con 0x7fc45015ef70 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc454c69640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc45015bbd0 con 0x7fc4500b3160 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48918/0 (socket says 172.21.15.70:48918) 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 -- 172.21.15.70:0/2427876785 learned_addr learned my addr 172.21.15.70:0/2427876785 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc44ed9d640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc45015ef70 0x7fc450153860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc44e59c640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4500b3160 0x7fc4501e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 -- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4500b3160 msgr2=0x7fc4501e5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4500b3160 0x7fc4501e5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 -- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc45015ef70 msgr2=0x7fc450153860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc45015ef70 0x7fc450153860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 -- 172.21.15.70:0/2427876785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4501f3570 con 0x7fc4500d0690 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.870+0000 7fc445d9b640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7c0 tx=0x7fc43800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.871+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438015070 con 0x7fc4500d0690 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.871+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4380079d0 con 0x7fc4500d0690 2024-04-15T15:17:12.876 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.871+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4501e50c0 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.871+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438007da0 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.871+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4501f3af0 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.872+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4501e54c0 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.872+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc438018070 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.872+0000 7fc4477fe640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 0x7fc43006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.873+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fc43808bc90 con 0x7fc4500d0690 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.873+0000 7fc44e59c640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 0x7fc43006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:12.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.873+0000 7fc44e59c640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 0x7fc43006ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc440036000 tx=0x7fc44003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:12.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:12.877+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fc438059770 con 0x7fc4500d0690 2024-04-15T15:17:13.056 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.055+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc4500689e0 con 0x7fc4500d0690 2024-04-15T15:17:13.057 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.056+0000 7fc4477fe640 1 -- 172.21.15.70:0/2427876785 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7fc43805da70 con 0x7fc4500d0690 2024-04-15T15:17:13.058 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:13.058 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 msgr2=0x7fc43006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 0x7fc43006ee40 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc440036000 tx=0x7fc44003c040 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 msgr2=0x7fc4500ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c7c0 tx=0x7fc43800cc80 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 shutdown_connections 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc45015ef70 0x7fc450153860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc43006c980 0x7fc43006ee40 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4500d0690 0x7fc4500ce4e0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 --2- 172.21.15.70:0/2427876785 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4500b3160 0x7fc4501e5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 >> 172.21.15.70:0/2427876785 conn(0x7fc45006be00 msgr2=0x7fc4500b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.060 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 shutdown_connections 2024-04-15T15:17:13.061 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.058+0000 7fc454c69640 1 -- 172.21.15.70:0/2427876785 wait complete. 2024-04-15T15:17:13.071 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T15:14:58.086322+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-15T15:16:23.468270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-15T15:16:24.186185+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-15T15:17:13.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-15T15:17:13.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.175+0000 7f4bc76d5640 1 Processor -- start 2024-04-15T15:17:13.178 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.176+0000 7f4bc76d5640 1 -- start start 2024-04-15T15:17:13.181 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc76d5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bc00cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc76d5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bc00b47a0 con 0x7f4bc015e700 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc76d5640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bc00b4a80 con 0x7f4bc00d2360 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc76d5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc00b53b0 con 0x7f4bc00d3270 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc5471640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4bc015e700 0x7f4bc00b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46412/0 (socket says 172.21.15.70:46412) 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc5471640 1 -- 172.21.15.70:0/2189843032 learned_addr learned my addr 172.21.15.70:0/2189843032 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc5c72640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bc00cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 316727925 0 0) 0x7f4bc00b47a0 con 0x7f4bc015e700 2024-04-15T15:17:13.182 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4bac003520 con 0x7f4bc015e700 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1488086662 0 0) 0x7f4bac003520 con 0x7f4bc015e700 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 msgr2=0x7f4bc00cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bc00cf370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 >> v1:172.21.15.96:6789/0 conn(0x7f4bc00d2360 legacy=0x7f4bc00d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc00b4d60 con 0x7f4bc015e700 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4bb40017f0 con 0x7f4bc015e700 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 >> v1:172.21.15.70:6789/0 conn(0x7f4bc015e700 legacy=0x7f4bc00b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bac0039e0 0x7f4bac003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bac004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc00b53b0 con 0x7f4bac004800 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac00baa0 con 0x7f4bac0039e0 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac00bca0 con 0x7f4bc00d3270 2024-04-15T15:17:13.183 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bac004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.177+0000 7f4bc5471640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bac0039e0 0x7f4bac003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc5c72640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 -- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 msgr2=0x7f4bac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00d3270 0x7f4bac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 -- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bac0039e0 msgr2=0x7f4bac003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bac0039e0 0x7f4bac003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 -- 172.21.15.70:0/2189843032 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac003520 con 0x7f4bac004800 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.178+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bac004c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001720 tx=0x7f4bac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c03010d226894ca server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc001370 con 0x7f4bac004800 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bbc0052d0 con 0x7f4bac004800 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bb3fff640 1 -- 172.21.15.70:0/2189843032 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc003830 con 0x7f4bac004800 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 -- 172.21.15.70:0/2189843032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 msgr2=0x7f4bac004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/2189843032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bac004c80 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc001720 tx=0x7f4bac005830 comp rx=0 tx=0).stop 2024-04-15T15:17:13.184 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc5c72640 1 -- 172.21.15.70:0/2189843032 reap_dead start 2024-04-15T15:17:13.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 -- 172.21.15.70:0/2189843032 shutdown_connections 2024-04-15T15:17:13.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 -- 172.21.15.70:0/2189843032 >> 172.21.15.70:0/2189843032 conn(0x7f4bc008bf60 msgr2=0x7f4bc00d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 -- 172.21.15.70:0/2189843032 shutdown_connections 2024-04-15T15:17:13.185 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.179+0000 7f4bc76d5640 1 -- 172.21.15.70:0/2189843032 wait complete. 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.180+0000 7f4bc76d5640 1 Processor -- start 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.180+0000 7f4bc76d5640 1 -- start start 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 0x7f4bc00cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00cc8d0 0x7f4bc01e7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc00d1d40 con 0x7f4bac004800 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4bc00b4450 con 0x7f4bc00b7130 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc76d5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4bc00d1ec0 con 0x7f4bc00cc8d0 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc5471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc5471640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48944/0 (socket says 172.21.15.70:48944) 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc5471640 1 -- 172.21.15.70:0/1145510567 learned_addr learned my addr 172.21.15.70:0/1145510567 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:13.189 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 0x7f4bc00cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.181+0000 7f4bc5c72640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00cc8d0 0x7f4bc01e7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 -- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00cc8d0 msgr2=0x7f4bc01e7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00cc8d0 0x7f4bc01e7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 -- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 msgr2=0x7f4bc00cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 0x7f4bc00cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 -- 172.21.15.70:0/1145510567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bc00b3c30 con 0x7f4bac004800 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 0x7f4bc00cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc5471640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400f820 tx=0x7f4bb400fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40072e0 con 0x7f4bac004800 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bb4018070 con 0x7f4bac004800 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bc00ccf50 con 0x7f4bac004800 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bb40057a0 con 0x7f4bac004800 2024-04-15T15:17:13.190 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.182+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bc01603b0 con 0x7f4bac004800 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.183+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bc00cccb0 con 0x7f4bac004800 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.184+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4bb4005940 con 0x7f4bac004800 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.184+0000 7f4bb1ffb640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 0x7f4ba406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.184+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 0x7f4ba406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.184+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f4bb408bdf0 con 0x7f4bac004800 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.185+0000 7f4bc4c70640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 0x7f4ba406ee40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc033000 tx=0x7f4bbc009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.191 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.189+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4bb401a310 con 0x7f4bac004800 2024-04-15T15:17:13.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.363+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4bc00689e0 con 0x7f4bac004800 2024-04-15T15:17:13.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.363+0000 7f4bb1ffb640 1 -- 172.21.15.70:0/1145510567 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) ==== 93+0+10 (secure 0 0 0) 0x7f4bb405a030 con 0x7f4bac004800 2024-04-15T15:17:13.365 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 1 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 msgr2=0x7f4ba406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 0x7f4ba406ee40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc033000 tx=0x7f4bbc009030 comp rx=0 tx=0).stop 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 msgr2=0x7f4bc015c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f4bb400f820 tx=0x7f4bb400fce0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 shutdown_connections 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4bc00cc8d0 0x7f4bc01e7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4bc00b7130 0x7f4bc00cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ba406c980 0x7f4ba406ee40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 --2- 172.21.15.70:0/1145510567 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4bac004800 0x7f4bc015c6c0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 >> 172.21.15.70:0/1145510567 conn(0x7f4bc008bf60 msgr2=0x7f4bc00d0660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.365+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 shutdown_connections 2024-04-15T15:17:13.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.366+0000 7f4bc76d5640 1 -- 172.21.15.70:0/1145510567 wait complete. 2024-04-15T15:17:13.379 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-15T15:17:13.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.483+0000 7f618e2ba640 1 Processor -- start 2024-04-15T15:17:13.485 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.483+0000 7f618e2ba640 1 -- start start 2024-04-15T15:17:13.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618e2ba640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 0x7f61880b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618e2ba640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61880cf800 0x7f61880cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618e2ba640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61880b1f40 con 0x7f618815fda0 2024-04-15T15:17:13.486 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618e2ba640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f61880d0150 con 0x7f61880cf800 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618e2ba640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f61880cda30 con 0x7f61880b1480 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61880cf800 0x7f61880cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f61880cf800 0x7f61880cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48954/0 (socket says 172.21.15.70:48954) 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 -- 172.21.15.70:0/2466718535 learned_addr learned my addr 172.21.15.70:0/2466718535 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f617ffff640 1 --2- 172.21.15.70:0/2466718535 >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 0x7f61880b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 -- 172.21.15.70:0/2466718535 >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 msgr2=0x7f61880b18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 --2- 172.21.15.70:0/2466718535 >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 0x7f61880b18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 -- 172.21.15.70:0/2466718535 >> v1:172.21.15.96:6789/0 conn(0x7f618815fda0 legacy=0x7f61880ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f618c857640 1 -- 172.21.15.70:0/2466718535 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61880cdd10 con 0x7f61880cf800 2024-04-15T15:17:13.487 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.484+0000 7f617ffff640 1 --2- 172.21.15.70:0/2466718535 >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 0x7f61880b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.485+0000 7f618c857640 1 --2- 172.21.15.70:0/2466718535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880cf800 0x7f61880cfc10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f618806a050 tx=0x7f618002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e282bea2617cd00 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.485+0000 7f61877fe640 1 -- 172.21.15.70:0/2466718535 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618002cb20 con 0x7f61880cf800 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.485+0000 7f61877fe640 1 -- 172.21.15.70:0/2466718535 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f618002ccc0 con 0x7f61880cf800 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.485+0000 7f61877fe640 1 -- 172.21.15.70:0/2466718535 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6180034610 con 0x7f61880cf800 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 -- 172.21.15.70:0/2466718535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880cf800 msgr2=0x7f61880cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2466718535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880cf800 0x7f61880cfc10 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f618806a050 tx=0x7f618002bf80 comp rx=0 tx=0).stop 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 -- 172.21.15.70:0/2466718535 shutdown_connections 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2466718535 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880cf800 0x7f61880cfc10 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2466718535 >> v2:172.21.15.96:3300/0 conn(0x7f61880b1480 0x7f61880b18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 -- 172.21.15.70:0/2466718535 >> 172.21.15.70:0/2466718535 conn(0x7f61880b6870 msgr2=0x7f61880b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 -- 172.21.15.70:0/2466718535 shutdown_connections 2024-04-15T15:17:13.489 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.486+0000 7f618e2ba640 1 -- 172.21.15.70:0/2466718535 wait complete. 2024-04-15T15:17:13.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.487+0000 7f618e2ba640 1 Processor -- start 2024-04-15T15:17:13.490 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.487+0000 7f618e2ba640 1 -- start start 2024-04-15T15:17:13.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61880cf800 0x7f6188155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f618815fda0 0x7f618806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6188160180 con 0x7f61880b1480 2024-04-15T15:17:13.492 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f61880ce2d0 con 0x7f61880cf800 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618e2ba640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f61880cef00 con 0x7f618815fda0 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48956/0 (socket says 172.21.15.70:48956) 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 -- 172.21.15.70:0/2440227292 learned_addr learned my addr 172.21.15.70:0/2440227292 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f617ffff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61880cf800 0x7f6188155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f618c857640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f618815fda0 0x7f618806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 -- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f618815fda0 msgr2=0x7f618806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f618815fda0 0x7f618806b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 -- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61880cf800 msgr2=0x7f6188155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61880cf800 0x7f6188155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 -- 172.21.15.70:0/2440227292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61880c9c50 con 0x7f61880b1480 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.488+0000 7f6187fff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f617800c3c0 tx=0x7f617800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.493 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.489+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61780052d0 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.489+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6178005490 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.489+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61880c6ea0 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.489+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6178007260 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.489+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61880ca1e0 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.490+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61880689e0 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.490+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6178007400 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.491+0000 7f61857fa640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 0x7f616806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.491+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f617808b130 con 0x7f61880b1480 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.491+0000 7f617ffff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 0x7f616806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.494 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.491+0000 7f617ffff640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 0x7f616806ee40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f61880d1230 tx=0x7f6170008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.497 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.495+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6178058db0 con 0x7f61880b1480 2024-04-15T15:17:13.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.669+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f61880b74e0 con 0x7f61880b1480 2024-04-15T15:17:13.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.669+0000 7f61857fa640 1 -- 172.21.15.70:0/2440227292 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) ==== 104+0+11 (secure 0 0 0) 0x7f617805d0b0 con 0x7f61880b1480 2024-04-15T15:17:13.671 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 32 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 msgr2=0x7f616806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 0x7f616806ee40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f61880d1230 tx=0x7f6170008040 comp rx=0 tx=0).stop 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 msgr2=0x7f61880cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f617800c3c0 tx=0x7f617800c880 comp rx=0 tx=0).stop 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 shutdown_connections 2024-04-15T15:17:13.673 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f618815fda0 0x7f618806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f61880cf800 0x7f6188155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f616806c980 0x7f616806ee40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 --2- 172.21.15.70:0/2440227292 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f61880b1480 0x7f61880cb710 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 >> 172.21.15.70:0/2440227292 conn(0x7f61880b6870 msgr2=0x7f6188070760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 shutdown_connections 2024-04-15T15:17:13.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.671+0000 7f618e2ba640 1 -- 172.21.15.70:0/2440227292 wait complete. 2024-04-15T15:17:13.684 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-15T15:17:13.790 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.789+0000 7fe08f39d640 1 Processor -- start 2024-04-15T15:17:13.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.789+0000 7fe08f39d640 1 -- start start 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08f39d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08f39d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 0x7fe0880d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08f39d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe088070720 con 0x7fe08809a9d0 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08f39d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe08809ba90 con 0x7fe08815fb30 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08f39d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe0880d08c0 con 0x7fe0880701f0 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:56386/0 (socket says 172.21.15.70:56386) 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08d93a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 0x7fe0880d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 -- 172.21.15.70:0/3192628987 learned_addr learned my addr 172.21.15.70:0/3192628987 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 -- 172.21.15.70:0/3192628987 >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 msgr2=0x7fe0880d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:13.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 --2- 172.21.15.70:0/3192628987 >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 0x7fe0880d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 -- 172.21.15.70:0/3192628987 >> v1:172.21.15.96:6789/0 conn(0x7fe08809a9d0 legacy=0x7fe08809adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08c938640 1 -- 172.21.15.70:0/3192628987 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe088070a90 con 0x7fe0880701f0 2024-04-15T15:17:13.793 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.790+0000 7fe08d93a640 1 --2- 172.21.15.70:0/3192628987 >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 0x7fe0880d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.791+0000 7fe08c938640 1 --2- 172.21.15.70:0/3192628987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0780098a0 tx=0x7fe07802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=538e1d486011ecc3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.791+0000 7fe07ffff640 1 -- 172.21.15.70:0/3192628987 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078034070 con 0x7fe0880701f0 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.791+0000 7fe07ffff640 1 -- 172.21.15.70:0/3192628987 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe078030840 con 0x7fe0880701f0 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.791+0000 7fe07ffff640 1 -- 172.21.15.70:0/3192628987 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078030c10 con 0x7fe0880701f0 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 -- 172.21.15.70:0/3192628987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 msgr2=0x7fe088160420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 --2- 172.21.15.70:0/3192628987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0780098a0 tx=0x7fe07802b980 comp rx=0 tx=0).stop 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 -- 172.21.15.70:0/3192628987 shutdown_connections 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 --2- 172.21.15.70:0/3192628987 >> v2:172.21.15.96:3300/0 conn(0x7fe08815fb30 0x7fe0880d0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 --2- 172.21.15.70:0/3192628987 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe0880701f0 0x7fe088160420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 -- 172.21.15.70:0/3192628987 >> 172.21.15.70:0/3192628987 conn(0x7fe0880d2460 msgr2=0x7fe0880d2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 -- 172.21.15.70:0/3192628987 shutdown_connections 2024-04-15T15:17:13.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.792+0000 7fe08f39d640 1 -- 172.21.15.70:0/3192628987 wait complete. 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.793+0000 7fe08f39d640 1 Processor -- start 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.793+0000 7fe08f39d640 1 -- start start 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe08809a9d0 0x7fe0880b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe08815fb30 0x7fe0880b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe08815ffc0 con 0x7fe0880701f0 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe088071050 con 0x7fe08815fb30 2024-04-15T15:17:13.797 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08f39d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe0880b1c60 con 0x7fe08809a9d0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48958/0 (socket says 172.21.15.70:48958) 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 -- 172.21.15.70:0/701142625 learned_addr learned my addr 172.21.15.70:0/701142625 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08c938640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe08809a9d0 0x7fe0880b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d93a640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe08815fb30 0x7fe0880b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 -- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe08809a9d0 msgr2=0x7fe0880b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe08809a9d0 0x7fe0880b76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 -- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe08815fb30 msgr2=0x7fe0880b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe08815fb30 0x7fe0880b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 -- 172.21.15.70:0/701142625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0880b5310 con 0x7fe0880701f0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe08d139640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c400 tx=0x7fe08400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0840052d0 con 0x7fe0880701f0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.794+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe084005490 con 0x7fe0880701f0 2024-04-15T15:17:13.798 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.795+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe08815d2d0 con 0x7fe0880701f0 2024-04-15T15:17:13.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.795+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe084007260 con 0x7fe0880701f0 2024-04-15T15:17:13.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.795+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0880ccbc0 con 0x7fe0880701f0 2024-04-15T15:17:13.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.796+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0880689e0 con 0x7fe0880701f0 2024-04-15T15:17:13.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.796+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe084007400 con 0x7fe0880701f0 2024-04-15T15:17:13.799 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.796+0000 7fe07dffb640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 0x7fe06c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:13.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.797+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fe08408b0e0 con 0x7fe0880701f0 2024-04-15T15:17:13.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.797+0000 7fe08c938640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 0x7fe06c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:13.800 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.797+0000 7fe08c938640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 0x7fe06c06ee40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe078036000 tx=0x7fe078008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:13.802 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.801+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe084058d60 con 0x7fe0880701f0 2024-04-15T15:17:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.981+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe0880ccd90 con 0x7fe0880701f0 2024-04-15T15:17:13.983 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.982+0000 7fe07dffb640 1 -- 172.21.15.70:0/701142625 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) ==== 100+0+11 (secure 0 0 0) 0x7fe08405d060 con 0x7fe0880701f0 2024-04-15T15:17:13.984 INFO:teuthology.orchestra.run.smithi070.stdout:pg_num: 64 2024-04-15T15:17:13.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 msgr2=0x7fe06c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 0x7fe06c06ee40 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe078036000 tx=0x7fe078008030 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 msgr2=0x7fe088154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fe08400c400 tx=0x7fe08400c800 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 shutdown_connections 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe08815fb30 0x7fe0880b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe08809a9d0 0x7fe0880b76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe06c06c980 0x7fe06c06ee40 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 --2- 172.21.15.70:0/701142625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe0880701f0 0x7fe088154680 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.983+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 >> 172.21.15.70:0/701142625 conn(0x7fe0880d2460 msgr2=0x7fe088162570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.984+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 shutdown_connections 2024-04-15T15:17:13.986 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:13.984+0000 7fe08f39d640 1 -- 172.21.15.70:0/701142625 wait complete. 2024-04-15T15:17:13.997 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:14.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.097+0000 7f209ece4640 1 Processor -- start 2024-04-15T15:17:14.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.097+0000 7f209ece4640 1 -- start start 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ece4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f209815fda0 0x7f20980ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ece4640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 0x7f20980cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ece4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20980b1f40 con 0x7f20980ced30 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ece4640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f20980d0150 con 0x7f209815fda0 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ece4640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f20980cda30 con 0x7f20980cf800 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ca80640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f209815fda0 0x7f20980ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ca80640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f209815fda0 0x7f20980ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48964/0 (socket says 172.21.15.70:48964) 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.098+0000 7f209ca80640 1 -- 172.21.15.70:0/4156478666 learned_addr learned my addr 172.21.15.70:0/4156478666 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209d281640 1 --2- 172.21.15.70:0/4156478666 >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 0x7f20980cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209ca80640 1 -- 172.21.15.70:0/4156478666 >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 msgr2=0x7f20980cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209ca80640 1 --2- 172.21.15.70:0/4156478666 >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 0x7f20980cfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209ca80640 1 -- 172.21.15.70:0/4156478666 >> v1:172.21.15.96:6789/0 conn(0x7f20980ced30 legacy=0x7f20980b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209ca80640 1 -- 172.21.15.70:0/4156478666 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20980cdd10 con 0x7f209815fda0 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209d281640 1 --2- 172.21.15.70:0/4156478666 >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 0x7f20980cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.099+0000 7f209ca80640 1 --2- 172.21.15.70:0/4156478666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f209815fda0 0x7f20980ce7f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f209806caa0 tx=0x7f208c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6779d10d3f59036 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f20977fe640 1 -- 172.21.15.70:0/4156478666 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c035020 con 0x7f209815fda0 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f20977fe640 1 -- 172.21.15.70:0/4156478666 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f208c031870 con 0x7f209815fda0 2024-04-15T15:17:14.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f20977fe640 1 -- 172.21.15.70:0/4156478666 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f208c031c40 con 0x7f209815fda0 2024-04-15T15:17:14.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 -- 172.21.15.70:0/4156478666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f209815fda0 msgr2=0x7f20980ce7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 --2- 172.21.15.70:0/4156478666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f209815fda0 0x7f20980ce7f0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f209806caa0 tx=0x7f208c02b910 comp rx=0 tx=0).stop 2024-04-15T15:17:14.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 -- 172.21.15.70:0/4156478666 shutdown_connections 2024-04-15T15:17:14.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 --2- 172.21.15.70:0/4156478666 >> v2:172.21.15.96:3300/0 conn(0x7f20980cf800 0x7f20980cfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 --2- 172.21.15.70:0/4156478666 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f209815fda0 0x7f20980ce7f0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 -- 172.21.15.70:0/4156478666 >> 172.21.15.70:0/4156478666 conn(0x7f20980b6870 msgr2=0x7f20980b6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 -- 172.21.15.70:0/4156478666 shutdown_connections 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.100+0000 7f209ece4640 1 -- 172.21.15.70:0/4156478666 wait complete. 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.101+0000 7f209ece4640 1 Processor -- start 2024-04-15T15:17:14.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.101+0000 7f209ece4640 1 -- start start 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 0x7f2098151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f209815fda0 0x7f20980b57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2098160180 con 0x7f20980cf800 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20980ce2d0 con 0x7f209815fda0 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f209ece4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f20980b0790 con 0x7f20980ced30 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f2097fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f2097fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48972/0 (socket says 172.21.15.70:48972) 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.102+0000 7f2097fff640 1 -- 172.21.15.70:0/661539547 learned_addr learned my addr 172.21.15.70:0/661539547 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:14.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f209d281640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f209815fda0 0x7f20980b57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f209ca80640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 0x7f2098151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 -- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 msgr2=0x7f2098151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 0x7f2098151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 -- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f209815fda0 msgr2=0x7f20980b57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f209815fda0 0x7f20980b57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 -- 172.21.15.70:0/661539547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20981501f0 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f209ca80640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 0x7f2098151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f2097fff640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f208400b3c0 tx=0x7f208400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2084011830 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2084016070 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20981504d0 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2084011c10 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.103+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2098150a90 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.104+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20980689e0 con 0x7f20980cf800 2024-04-15T15:17:14.107 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.104+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f208401b460 con 0x7f20980cf800 2024-04-15T15:17:14.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.105+0000 7f20957fa640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 0x7f207c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.105+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f208408b730 con 0x7f20980cf800 2024-04-15T15:17:14.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.105+0000 7f209ca80640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 0x7f207c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.105+0000 7f209ca80640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 0x7f207c06ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f208c036000 tx=0x7f208c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.109+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f2084020020 con 0x7f20980cf800 2024-04-15T15:17:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.319+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f20980b1330 con 0x7f20980cf800 2024-04-15T15:17:14.321 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.319+0000 7f20957fa640 1 -- 172.21.15.70:0/661539547 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5179 (secure 0 0 0) 0x7f2084059a00 con 0x7f20980cf800 2024-04-15T15:17:14.322 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:14.322 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:03.637138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5518,"qdb_cluster":[5518]},"id":2}]} 2024-04-15T15:17:14.322 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 20 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.321+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 msgr2=0x7f207c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.321+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 0x7f207c06ee40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f208c036000 tx=0x7f208c03c040 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.321+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 msgr2=0x7f2098144eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.321+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f208400b3c0 tx=0x7f208400b880 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 shutdown_connections 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f209815fda0 0x7f20980b57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f207c06c980 0x7f207c06ee40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20980cf800 0x7f2098144eb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 --2- 172.21.15.70:0/661539547 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20980ced30 0x7f2098151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 >> 172.21.15.70:0/661539547 conn(0x7f20980b6870 msgr2=0x7f20980d1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 shutdown_connections 2024-04-15T15:17:14.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.322+0000 7f209ece4640 1 -- 172.21.15.70:0/661539547 wait complete. 2024-04-15T15:17:14.335 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:14.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.435+0000 7fa21e077640 1 Processor -- start 2024-04-15T15:17:14.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.435+0000 7fa21e077640 1 -- start start 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa21e077640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa21815eb00 0x7fa2180ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa21e077640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2180cbd10 con 0x7fa21806c0f0 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa21e077640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2180cbff0 con 0x7fa2180b0560 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa21e077640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa21815eee0 con 0x7fa21815eb00 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa217fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa21806c0f0 0x7fa2180ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46424/0 (socket says 172.21.15.70:46424) 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.436+0000 7fa217fff640 1 -- 172.21.15.70:0/4123869187 learned_addr learned my addr 172.21.15.70:0/4123869187 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 172955296 0 0) 0x7fa2180cbd10 con 0x7fa21806c0f0 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa2177fe640 1 --2- 172.21.15.70:0/4123869187 >> v2:172.21.15.96:3300/0 conn(0x7fa21815eb00 0x7fa2180ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa204003520 con 0x7fa21806c0f0 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2123728809 0 0) 0x7fa204003520 con 0x7fa21806c0f0 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 >> v2:172.21.15.96:3300/0 conn(0x7fa21815eb00 msgr2=0x7fa2180ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 --2- 172.21.15.70:0/4123869187 >> v2:172.21.15.96:3300/0 conn(0x7fa21815eb00 0x7fa2180ae2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 >> v1:172.21.15.96:6789/0 conn(0x7fa2180b0560 legacy=0x7fa21806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2180cc2d0 con 0x7fa21806c0f0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fa2100019f0 con 0x7fa21806c0f0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 >> v1:172.21.15.70:6789/0 conn(0x7fa21806c0f0 legacy=0x7fa2180ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2040039b0 0x7fa204003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa204004800 0x7fa204004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa20400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa21815eee0 con 0x7fa2040051c0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa20400c060 con 0x7fa204004800 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa20400c260 con 0x7fa2040039b0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa20400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa20effd640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa204004800 0x7fa204004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa2177fe640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2040039b0 0x7fa204003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 -- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2040039b0 msgr2=0x7fa204003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2040039b0 0x7fa204003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 -- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa204004800 msgr2=0x7fa204004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa204004800 0x7fa204004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.437+0000 7fa217fff640 1 -- 172.21.15.70:0/4123869187 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa204003520 con 0x7fa2040051c0 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.438+0000 7fa217fff640 1 -- 172.21.15.70:0/4123869187 reap_dead start 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.438+0000 7fa217fff640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa20400b9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa218073c40 tx=0x7fa204005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=594f391c5b2179f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.438+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21000a4a0 con 0x7fa2040051c0 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.438+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa210034650 con 0x7fa2040051c0 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.438+0000 7fa216ffd640 1 -- 172.21.15.70:0/4123869187 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa210034a40 con 0x7fa2040051c0 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 -- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 msgr2=0x7fa20400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa20400b9c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fa218073c40 tx=0x7fa204005830 comp rx=0 tx=0).stop 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 -- 172.21.15.70:0/4123869187 shutdown_connections 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 --2- 172.21.15.70:0/4123869187 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa20400b9c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 -- 172.21.15.70:0/4123869187 >> 172.21.15.70:0/4123869187 conn(0x7fa21809a9d0 msgr2=0x7fa2180d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 -- 172.21.15.70:0/4123869187 shutdown_connections 2024-04-15T15:17:14.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.439+0000 7fa21e077640 1 -- 172.21.15.70:0/4123869187 wait complete. 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.440+0000 7fa21e077640 1 Processor -- start 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.440+0000 7fa21e077640 1 -- start start 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa21814c360 0x7fa2180c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa21814e8e0 0x7fa21814ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.446 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2180cec60 con 0x7fa2040051c0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2180cc890 con 0x7fa21814e8e0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa21e077640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa2180cede0 con 0x7fa21814c360 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48990/0 (socket says 172.21.15.70:48990) 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 -- 172.21.15.70:0/463104221 learned_addr learned my addr 172.21.15.70:0/463104221 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa20effd640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa21814c360 0x7fa2180c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa217fff640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa21814e8e0 0x7fa21814ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 -- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa21814c360 msgr2=0x7fa2180c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa21814c360 0x7fa2180c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 -- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa21814e8e0 msgr2=0x7fa21814ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa21814e8e0 0x7fa21814ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 -- 172.21.15.70:0/463104221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2180cb380 con 0x7fa2040051c0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa2177fe640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa208011540 tx=0x7fa208011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.441+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa208015930 con 0x7fa2040051c0 2024-04-15T15:17:14.447 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.442+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa208015ad0 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.442+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2180b5e40 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.442+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa20800f460 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.442+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2180b6400 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.443+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2180690f0 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.443+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa20800f600 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.443+0000 7fa214ff9640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 0x7fa1f806ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.443+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7fa20808c5b0 con 0x7fa2040051c0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.443+0000 7fa20effd640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 0x7fa1f806ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.448 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.444+0000 7fa20effd640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 0x7fa1f806ee70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa21814c760 tx=0x7fa200009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.449 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.448+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa208019030 con 0x7fa2040051c0 2024-04-15T15:17:14.623 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.622+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2180cea20 con 0x7fa2040051c0 2024-04-15T15:17:14.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.623+0000 7fa214ff9640 1 -- 172.21.15.70:0/463104221 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7fa20805a880 con 0x7fa2040051c0 2024-04-15T15:17:14.625 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:14.625 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.624+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 msgr2=0x7fa1f806ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.624+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 0x7fa1f806ee70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fa21814c760 tx=0x7fa200009040 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 msgr2=0x7fa2180b6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fa208011540 tx=0x7fa208011a00 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 shutdown_connections 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa21814e8e0 0x7fa21814ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa21814c360 0x7fa2180c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa1f806c9b0 0x7fa1f806ee70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 --2- 172.21.15.70:0/463104221 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2040051c0 0x7fa2180b6920 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 >> 172.21.15.70:0/463104221 conn(0x7fa21809a9d0 msgr2=0x7fa2180d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 shutdown_connections 2024-04-15T15:17:14.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.625+0000 7fa21e077640 1 -- 172.21.15.70:0/463104221 wait complete. 2024-04-15T15:17:14.638 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:17:14.639 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:14.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.741+0000 7f74bdd9d640 1 Processor -- start 2024-04-15T15:17:14.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.741+0000 7f74bdd9d640 1 -- start start 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74bdd9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f74b8072d80 0x7f74b8071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74bdd9d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f74b80af3e0 0x7f74b80af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74bdd9d640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74b815db40 con 0x7f74b80d0690 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74bdd9d640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f74b815cfc0 con 0x7f74b8072d80 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74bdd9d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f74b815d2a0 con 0x7f74b80af3e0 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f74b8072d80 0x7f74b8071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f74b8072d80 0x7f74b8071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:48996/0 (socket says 172.21.15.70:48996) 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 -- 172.21.15.70:0/974971669 learned_addr learned my addr 172.21.15.70:0/974971669 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 -- 172.21.15.70:0/974971669 >> v2:172.21.15.96:3300/0 conn(0x7f74b80af3e0 msgr2=0x7f74b80af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 --2- 172.21.15.70:0/974971669 >> v2:172.21.15.96:3300/0 conn(0x7f74b80af3e0 0x7f74b80af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 -- 172.21.15.70:0/974971669 >> v1:172.21.15.96:6789/0 conn(0x7f74b80d0690 legacy=0x7f74b806e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:14.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.742+0000 7f74b77fe640 1 -- 172.21.15.70:0/974971669 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b815d580 con 0x7f74b8072d80 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74b77fe640 1 --2- 172.21.15.70:0/974971669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b8072d80 0x7f74b8071d10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f74b80d1240 tx=0x7f74a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1dfedba61a124a server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74b67fc640 1 -- 172.21.15.70:0/974971669 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a802e070 con 0x7f74b8072d80 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74b67fc640 1 -- 172.21.15.70:0/974971669 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74a8037070 con 0x7f74b8072d80 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74b67fc640 1 -- 172.21.15.70:0/974971669 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a8031a00 con 0x7f74b8072d80 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 -- 172.21.15.70:0/974971669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b8072d80 msgr2=0x7f74b8071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/974971669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b8072d80 0x7f74b8071d10 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f74b80d1240 tx=0x7f74a802b880 comp rx=0 tx=0).stop 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 -- 172.21.15.70:0/974971669 shutdown_connections 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/974971669 >> v2:172.21.15.96:3300/0 conn(0x7f74b80af3e0 0x7f74b80af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/974971669 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b8072d80 0x7f74b8071d10 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 -- 172.21.15.70:0/974971669 >> 172.21.15.70:0/974971669 conn(0x7f74b806be00 msgr2=0x7f74b806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.743+0000 7f74bdd9d640 1 -- 172.21.15.70:0/974971669 shutdown_connections 2024-04-15T15:17:14.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.744+0000 7f74bdd9d640 1 -- 172.21.15.70:0/974971669 wait complete. 2024-04-15T15:17:14.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.745+0000 7f74bdd9d640 1 Processor -- start 2024-04-15T15:17:14.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.745+0000 7f74bdd9d640 1 -- start start 2024-04-15T15:17:14.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.745+0000 7f74bdd9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f74b8072d80 0x7f74b81554a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.745+0000 7f74bdd9d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74b80af3e0 0x7f74b80cec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f74b815cd10 con 0x7f74b80d0690 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f74b815cb90 con 0x7f74b80af3e0 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f74b80d0b00 con 0x7f74b8072d80 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49012/0 (socket says 172.21.15.70:49012) 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 -- 172.21.15.70:0/3980053272 learned_addr learned my addr 172.21.15.70:0/3980053272 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b77fe640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f74b8072d80 0x7f74b81554a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 -- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f74b8072d80 msgr2=0x7f74b81554a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f74b8072d80 0x7f74b81554a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 -- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74b80af3e0 msgr2=0x7f74b80cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74b80af3e0 0x7f74b80cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 -- 172.21.15.70:0/3980053272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b8158140 con 0x7f74b80d0690 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74b7fff640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f74b000b990 tx=0x7f74b000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0019070 con 0x7f74b80d0690 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74b000cba0 con 0x7f74b80d0690 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b8158970 con 0x7f74b80d0690 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b0014440 con 0x7f74b80d0690 2024-04-15T15:17:14.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.746+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b8158f30 con 0x7f74b80d0690 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.747+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b8159130 con 0x7f74b80d0690 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.748+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f74b001c070 con 0x7f74b80d0690 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.748+0000 7f749ffff640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 0x7f74a406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.748+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f74b008bb50 con 0x7f74b80d0690 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.748+0000 7f74b77fe640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 0x7f74a406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:14.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.749+0000 7f74b77fe640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 0x7f74a406ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036000 tx=0x7f74a8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:14.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.752+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f74b8159130 con 0x7f74b80d0690 2024-04-15T15:17:14.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.965+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74b80689e0 con 0x7f74b80d0690 2024-04-15T15:17:14.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.965+0000 7f749ffff640 1 -- 172.21.15.70:0/3980053272 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5179 (secure 0 0 0) 0x7f74b0059d90 con 0x7f74b80d0690 2024-04-15T15:17:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:14.970 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:03.637138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5518,"qdb_cluster":[5518]},"id":2}]} 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 20 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 msgr2=0x7f74a406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 0x7f74a406ee40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f74a8036000 tx=0x7f74a8008030 comp rx=0 tx=0).stop 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 msgr2=0x7f74b80cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f74b000b990 tx=0x7f74b000be50 comp rx=0 tx=0).stop 2024-04-15T15:17:14.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 shutdown_connections 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f74a406c980 0x7f74a406ee40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f74b80d0690 0x7f74b80cf1d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f74b80af3e0 0x7f74b80cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 --2- 172.21.15.70:0/3980053272 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f74b8072d80 0x7f74b81554a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 >> 172.21.15.70:0/3980053272 conn(0x7f74b806be00 msgr2=0x7f74b8074d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 shutdown_connections 2024-04-15T15:17:14.972 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:14.967+0000 7f74bdd9d640 1 -- 172.21.15.70:0/3980053272 wait complete. 2024-04-15T15:17:14.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:15.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.082+0000 7f5231e4e640 1 Processor -- start 2024-04-15T15:17:15.084 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.082+0000 7f5231e4e640 1 -- start start 2024-04-15T15:17:15.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f5231e4e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f522c072d80 0x7f522c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f5231e4e640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f522c0af3e0 0x7f522c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f5231e4e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f522c15db40 con 0x7f522c0d0690 2024-04-15T15:17:15.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f5231e4e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f522c15cfc0 con 0x7f522c072d80 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f5231e4e640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f522c15d2a0 con 0x7f522c0af3e0 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522b7fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f522c072d80 0x7f522c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f522c0d0690 0x7f522c06e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46428/0 (socket says 172.21.15.70:46428) 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522bfff640 1 -- 172.21.15.70:0/629957415 learned_addr learned my addr 172.21.15.70:0/629957415 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522affd640 1 --2- 172.21.15.70:0/629957415 >> v2:172.21.15.96:3300/0 conn(0x7f522c0af3e0 0x7f522c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522b7fe640 1 -- 172.21.15.70:0/629957415 >> v2:172.21.15.96:3300/0 conn(0x7f522c0af3e0 msgr2=0x7f522c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522b7fe640 1 --2- 172.21.15.70:0/629957415 >> v2:172.21.15.96:3300/0 conn(0x7f522c0af3e0 0x7f522c0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522b7fe640 1 -- 172.21.15.70:0/629957415 >> v1:172.21.15.70:6789/0 conn(0x7f522c0d0690 legacy=0x7f522c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522a7fc640 1 -- 172.21.15.70:0/629957415 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290754407 0 0) 0x7f522c15db40 con 0x7f522c0d0690 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.083+0000 7f522b7fe640 1 -- 172.21.15.70:0/629957415 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c15d580 con 0x7f522c072d80 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.084+0000 7f522b7fe640 1 --2- 172.21.15.70:0/629957415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c071d10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f522c0d1240 tx=0x7f521c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dda73440a22d695 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.084+0000 7f522a7fc640 1 -- 172.21.15.70:0/629957415 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c0315a0 con 0x7f522c072d80 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.084+0000 7f522a7fc640 1 -- 172.21.15.70:0/629957415 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c037070 con 0x7f522c072d80 2024-04-15T15:17:15.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.084+0000 7f522a7fc640 1 -- 172.21.15.70:0/629957415 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c031960 con 0x7f522c072d80 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.084+0000 7f5231e4e640 1 -- 172.21.15.70:0/629957415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 msgr2=0x7f522c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 --2- 172.21.15.70:0/629957415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c071d10 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f522c0d1240 tx=0x7f521c02b840 comp rx=0 tx=0).stop 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 -- 172.21.15.70:0/629957415 shutdown_connections 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 --2- 172.21.15.70:0/629957415 >> v2:172.21.15.96:3300/0 conn(0x7f522c0af3e0 0x7f522c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 --2- 172.21.15.70:0/629957415 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c071d10 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 -- 172.21.15.70:0/629957415 >> 172.21.15.70:0/629957415 conn(0x7f522c06be00 msgr2=0x7f522c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 -- 172.21.15.70:0/629957415 shutdown_connections 2024-04-15T15:17:15.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.085+0000 7f5231e4e640 1 -- 172.21.15.70:0/629957415 wait complete. 2024-04-15T15:17:15.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.086+0000 7f5231e4e640 1 Processor -- start 2024-04-15T15:17:15.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.086+0000 7f5231e4e640 1 -- start start 2024-04-15T15:17:15.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 0x7f522c0b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 0x7f522c155310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f522c15cd10 con 0x7f522c072d80 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f522c15cb90 con 0x7f522c0d0690 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f522c0d0b00 con 0x7f522c0af3e0 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49018/0 (socket says 172.21.15.70:49018) 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 -- 172.21.15.70:0/3666519241 learned_addr learned my addr 172.21.15.70:0/3666519241 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522affd640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 0x7f522c0b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522bfff640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 0x7f522c155310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 -- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 msgr2=0x7f522c0b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 0x7f522c0b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 -- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 msgr2=0x7f522c155310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 0x7f522c155310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 -- 172.21.15.70:0/3666519241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f522c0b3e90 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522affd640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 0x7f522c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522bfff640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 0x7f522c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f522b7fe640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f521c009890 tx=0x7f521c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c032840 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f521c0329e0 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f522c155850 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f521c032dd0 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.087+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f522c0b43f0 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.088+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f522c0689e0 con 0x7f522c072d80 2024-04-15T15:17:15.092 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.089+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f521c03b3f0 con 0x7f522c072d80 2024-04-15T15:17:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.089+0000 7f5213fff640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 0x7f521806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.089+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f521c02f030 con 0x7f522c072d80 2024-04-15T15:17:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.089+0000 7f522affd640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 0x7f521806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.090+0000 7f522affd640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 0x7f521806ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f522c0c98c0 tx=0x7f5220008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.094+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f522c0b3e90 con 0x7f522c072d80 2024-04-15T15:17:15.308 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.306+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f522c0728c0 con 0x7f522c072d80 2024-04-15T15:17:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.307+0000 7f5213fff640 1 -- 172.21.15.70:0/3666519241 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5179 (secure 0 0 0) 0x7f521c040070 con 0x7f522c072d80 2024-04-15T15:17:15.309 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:15.309 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-15T15:16:25.228602+0000","modified":"2024-04-15T15:17:03.637138+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5518},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5518":{"gid":5518,"name":"d","rank":0,"incarnation":17,"state":"up:active","state_seq":4,"addr":"172.21.15.96:6835/1131645259","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":1131645259},{"type":"v1","addr":"172.21.15.96:6835","nonce":1131645259}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5518,"qdb_cluster":[5518]},"id":2}]} 2024-04-15T15:17:15.309 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 20 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 msgr2=0x7f521806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 0x7f521806ee40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f522c0c98c0 tx=0x7f5220008040 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 msgr2=0x7f522c1f4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f521c009890 tx=0x7f521c0046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 shutdown_connections 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f522c0d0690 0x7f522c155310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f522c0af3e0 0x7f522c0b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f521806c980 0x7f521806ee40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 --2- 172.21.15.70:0/3666519241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f522c072d80 0x7f522c1f4440 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 >> 172.21.15.70:0/3666519241 conn(0x7f522c06be00 msgr2=0x7f522c071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 shutdown_connections 2024-04-15T15:17:15.312 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.309+0000 7f5231e4e640 1 -- 172.21.15.70:0/3666519241 wait complete. 2024-04-15T15:17:15.322 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.423+0000 7f1c90ddb640 1 Processor -- start 2024-04-15T15:17:15.425 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.423+0000 7f1c90ddb640 1 -- start start 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c90ddb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 0x7f1c8c0d0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c90ddb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 0x7f1c8c0d3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c90ddb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c0d1e60 con 0x7f1c8c0d1400 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c90ddb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f1c8c0d3550 con 0x7f1c8c15e9a0 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c90ddb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c0d18a0 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c89d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f1c8c0d1400 0x7f1c8c0cf210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46440/0 (socket says 172.21.15.70:46440) 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.424+0000 7f1c89d9b640 1 -- 172.21.15.70:0/881135758 learned_addr learned my addr 172.21.15.70:0/881135758 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8ad9d640 1 --2- 172.21.15.70:0/881135758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 0x7f1c8c0d3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8a59c640 1 --2- 172.21.15.70:0/881135758 >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 0x7f1c8c0d0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8959a640 1 -- 172.21.15.70:0/881135758 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 834671787 0 0) 0x7f1c8c0d1e60 con 0x7f1c8c0d1400 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8959a640 1 -- 172.21.15.70:0/881135758 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c78003520 con 0x7f1c8c0d1400 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8ad9d640 1 -- 172.21.15.70:0/881135758 >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 msgr2=0x7f1c8c0d0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8ad9d640 1 --2- 172.21.15.70:0/881135758 >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 0x7f1c8c0d0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8ad9d640 1 -- 172.21.15.70:0/881135758 >> v1:172.21.15.70:6789/0 conn(0x7f1c8c0d1400 legacy=0x7f1c8c0cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8ad9d640 1 -- 172.21.15.70:0/881135758 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c0d1b80 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.425+0000 7f1c8a59c640 1 --2- 172.21.15.70:0/881135758 >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 0x7f1c8c0d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c8ad9d640 1 --2- 172.21.15.70:0/881135758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 0x7f1c8c0d3010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d90 tx=0x7f1c8402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e5c0f12a253d7c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c8959a640 1 -- 172.21.15.70:0/881135758 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84039070 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c8959a640 1 -- 172.21.15.70:0/881135758 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8402cc50 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c8959a640 1 -- 172.21.15.70:0/881135758 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84034440 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c90ddb640 1 -- 172.21.15.70:0/881135758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 msgr2=0x7f1c8c0d3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.426+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/881135758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 0x7f1c8c0d3010 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d90 tx=0x7f1c8402bea0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 -- 172.21.15.70:0/881135758 shutdown_connections 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/881135758 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c0d2ba0 0x7f1c8c0d3010 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.429 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/881135758 >> v2:172.21.15.96:3300/0 conn(0x7f1c8c15e9a0 0x7f1c8c0d0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 -- 172.21.15.70:0/881135758 >> 172.21.15.70:0/881135758 conn(0x7f1c8c09d950 msgr2=0x7f1c8c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 -- 172.21.15.70:0/881135758 shutdown_connections 2024-04-15T15:17:15.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.427+0000 7f1c90ddb640 1 -- 172.21.15.70:0/881135758 wait complete. 2024-04-15T15:17:15.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.428+0000 7f1c90ddb640 1 Processor -- start 2024-04-15T15:17:15.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.428+0000 7f1c90ddb640 1 -- start start 2024-04-15T15:17:15.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 0x7f1c8c15c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 0x7f1c8c160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c0cf920 con 0x7f1c8c0d1400 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c06d2d0 con 0x7f1c8c0d2ba0 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c0d2360 con 0x7f1c8c15e9a0 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49028/0 (socket says 172.21.15.70:49028) 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 -- 172.21.15.70:0/1674677647 learned_addr learned my addr 172.21.15.70:0/1674677647 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c89d9b640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 0x7f1c8c15c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8ad9d640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 0x7f1c8c160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 -- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 msgr2=0x7f1c8c160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 0x7f1c8c160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 -- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 msgr2=0x7f1c8c15c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 0x7f1c8c15c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 -- 172.21.15.70:0/1674677647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c161500 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8ad9d640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 0x7f1c8c160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c89d9b640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 0x7f1c8c15c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c8a59c640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00e990 tx=0x7f1c7c00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c017070 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c00c9e0 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c15fd70 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c00cdb0 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.429+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c160330 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.430+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c0690f0 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.431+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f1c7c01a070 con 0x7f1c8c0d1400 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.431+0000 7f1c82ffd640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 0x7f1c6c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.432+0000 7f1c89d9b640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 0x7f1c6c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.432+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f1c7c08bf60 con 0x7f1c8c0d1400 2024-04-15T15:17:15.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.432+0000 7f1c89d9b640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 0x7f1c6c06ee40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c09dde0 tx=0x7f1c74009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.436+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f1c7c05a230 con 0x7f1c8c0d1400 2024-04-15T15:17:15.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.613+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c8c06cd30 con 0x7f1c8c0d1400 2024-04-15T15:17:15.616 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.614+0000 7f1c82ffd640 1 -- 172.21.15.70:0/1674677647 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) ==== 74+0+17003 (secure 0 0 0) 0x7f1c7c059b30 con 0x7f1c8c0d1400 2024-04-15T15:17:15.616 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:15.616 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":42,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:00.083667+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-15T15:16:23.468270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-15T15:16:24.186185+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:15.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 msgr2=0x7f1c6c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 0x7f1c6c06ee40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c09dde0 tx=0x7f1c74009040 comp rx=0 tx=0).stop 2024-04-15T15:17:15.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 msgr2=0x7f1c8c15f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c00e990 tx=0x7f1c7c00ee50 comp rx=0 tx=0).stop 2024-04-15T15:17:15.618 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 shutdown_connections 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f1c8c15e9a0 0x7f1c8c160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f1c8c0d2ba0 0x7f1c8c15c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f1c6c06c980 0x7f1c6c06ee40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 --2- 172.21.15.70:0/1674677647 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c8c0d1400 0x7f1c8c15f790 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 >> 172.21.15.70:0/1674677647 conn(0x7f1c8c09d950 msgr2=0x7f1c8c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 shutdown_connections 2024-04-15T15:17:15.619 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.616+0000 7f1c90ddb640 1 -- 172.21.15.70:0/1674677647 wait complete. 2024-04-15T15:17:15.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-15T15:17:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.732+0000 7f99f6a4a640 1 Processor -- start 2024-04-15T15:17:15.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.732+0000 7f99f6a4a640 1 -- start start 2024-04-15T15:17:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f6a4a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 0x7f99f00d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f6a4a640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f99f00ae280 0x7f99f00ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f6a4a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99f00d0fb0 con 0x7f99f00b3160 2024-04-15T15:17:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f6a4a640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f99f015bda0 con 0x7f99f00ae280 2024-04-15T15:17:15.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f6a4a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f99f015cb50 con 0x7f99f00d0690 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f4fe7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f99f00ae280 0x7f99f00ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f4fe7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f99f00ae280 0x7f99f00ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49030/0 (socket says 172.21.15.70:49030) 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99f4fe7640 1 -- 172.21.15.70:0/3851213890 learned_addr learned my addr 172.21.15.70:0/3851213890 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.733+0000 7f99effff640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 0x7f99f00d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99f4fe7640 1 -- 172.21.15.70:0/3851213890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 msgr2=0x7f99f00d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99f4fe7640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 0x7f99f00d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99f4fe7640 1 -- 172.21.15.70:0/3851213890 >> v1:172.21.15.96:6789/0 conn(0x7f99f00b3160 legacy=0x7f99f015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99f4fe7640 1 -- 172.21.15.70:0/3851213890 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f015bfd0 con 0x7f99f00ae280 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99effff640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 0x7f99f00d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.734+0000 7f99f4fe7640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f00ae6e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f99e800d150 tx=0x7f99e802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcfd72f8931c1a26 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99eeffd640 1 -- 172.21.15.70:0/3851213890 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e800b0d0 con 0x7f99f00ae280 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99eeffd640 1 -- 172.21.15.70:0/3851213890 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e800b270 con 0x7f99f00ae280 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99eeffd640 1 -- 172.21.15.70:0/3851213890 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e8009600 con 0x7f99f00ae280 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 -- 172.21.15.70:0/3851213890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 msgr2=0x7f99f00ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f00ae6e0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f99e800d150 tx=0x7f99e802e160 comp rx=0 tx=0).stop 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 -- 172.21.15.70:0/3851213890 shutdown_connections 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f00ae6e0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/3851213890 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00d0690 0x7f99f00d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 -- 172.21.15.70:0/3851213890 >> 172.21.15.70:0/3851213890 conn(0x7f99f006be00 msgr2=0x7f99f006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 -- 172.21.15.70:0/3851213890 shutdown_connections 2024-04-15T15:17:15.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.735+0000 7f99f6a4a640 1 -- 172.21.15.70:0/3851213890 wait complete. 2024-04-15T15:17:15.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.736+0000 7f99f6a4a640 1 Processor -- start 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.736+0000 7f99f6a4a640 1 -- start start 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00b3160 0x7f99f0146460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 0x7f99f0147ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f99f015c870 con 0x7f99f00ae280 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f99f015c590 con 0x7f99f00d0690 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99f6a4a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f99f00b4fb0 con 0x7f99f00b3160 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99effff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49044/0 (socket says 172.21.15.70:49044) 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99effff640 1 -- 172.21.15.70:0/1862223084 learned_addr learned my addr 172.21.15.70:0/1862223084 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99effff640 1 -- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00b3160 msgr2=0x7f99f0146460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.743 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.737+0000 7f99ef7fe640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00b3160 0x7f99f0146460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99effff640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00b3160 0x7f99f0146460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99f4fe7640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 0x7f99f0147ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99effff640 1 -- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 msgr2=0x7f99f0147ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99effff640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 0x7f99f0147ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99effff640 1 -- 172.21.15.70:0/1862223084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f0146c00 con 0x7f99f00ae280 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99f4fe7640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 0x7f99f0147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99effff640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f99e000e7e0 tx=0x7f99e000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e000f040 con 0x7f99f00ae280 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99e0004030 con 0x7f99f00ae280 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99e0004400 con 0x7f99f00ae280 2024-04-15T15:17:15.744 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f0146ee0 con 0x7f99f00ae280 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.738+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f01474a0 con 0x7f99f00ae280 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.739+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f00689e0 con 0x7f99f00ae280 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.739+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f99e000c6d0 con 0x7f99f00ae280 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.740+0000 7f99ecff9640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 0x7f99dc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.740+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f99e008bc60 con 0x7f99f00ae280 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.740+0000 7f99ef7fe640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 0x7f99dc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:15.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.740+0000 7f99ef7fe640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 0x7f99dc06ee40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f99f00b0fa0 tx=0x7f99e4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:15.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.744+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f99e0017020 con 0x7f99f00ae280 2024-04-15T15:17:15.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.956+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f99f00739e0 con 0x7f99f00ae280 2024-04-15T15:17:15.958 INFO:tasks.ceph.mon.a.smithi070.stderr:2024-04-15T15:17:15.957+0000 7fdd5b7bd640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-15T15:17:15.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.958+0000 7f99ecff9640 1 -- 172.21.15.70:0/1862223084 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) ==== 161+0+0 (secure 0 0 0) 0x7f99e0059e20 con 0x7f99f00ae280 2024-04-15T15:17:15.960 INFO:teuthology.orchestra.run.smithi070.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-15T15:17:15.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 msgr2=0x7f99dc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 0x7f99dc06ee40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f99f00b0fa0 tx=0x7f99e4009040 comp rx=0 tx=0).stop 2024-04-15T15:17:15.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 msgr2=0x7f99f0148f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f99e000e7e0 tx=0x7f99e000eca0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 shutdown_connections 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f99f00d0690 0x7f99f0147ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f99f00b3160 0x7f99f0146460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f99dc06c980 0x7f99dc06ee40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 --2- 172.21.15.70:0/1862223084 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99f00ae280 0x7f99f0148f90 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 >> 172.21.15.70:0/1862223084 conn(0x7f99f006be00 msgr2=0x7f99f00af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.960+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 shutdown_connections 2024-04-15T15:17:15.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:15.961+0000 7f99f6a4a640 1 -- 172.21.15.70:0/1862223084 wait complete. 2024-04-15T15:17:15.967 INFO:tasks.ceph.mds.d.smithi096.stderr: -1282> 2024-04-15T15:17:05.636+0000 7f9636f21640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-15T15:17:15.973 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-15T15:17:15.994 INFO:tasks.ceph.mds.d.smithi096.stdout:starting mds.d at 2024-04-15T15:17:16.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.077+0000 7f6c654b2640 1 Processor -- start 2024-04-15T15:17:16.079 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.077+0000 7f6c654b2640 1 -- start start 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c654b2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 0x7f6c600cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c654b2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c600b5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c654b2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c600d0350 con 0x7f6c6015c8d0 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c654b2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f6c600b5b00 con 0x7f6c6009b0e0 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c654b2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6c600ce100 con 0x7f6c6009b650 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c57fff640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 0x7f6c600cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5effd640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7f6c6015c8d0 0x7f6c6009a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:40306/0 (socket says 172.21.15.70:40306) 2024-04-15T15:17:16.080 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5f7fe640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c600b5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5effd640 1 -- 172.21.15.70:0/1381943225 learned_addr learned my addr 172.21.15.70:0/1381943225 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5f7fe640 1 -- 172.21.15.70:0/1381943225 >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 msgr2=0x7f6c600cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5f7fe640 1 --2- 172.21.15.70:0/1381943225 >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 0x7f6c600cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5f7fe640 1 -- 172.21.15.70:0/1381943225 >> v1:172.21.15.96:6789/0 conn(0x7f6c6015c8d0 legacy=0x7f6c6009a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5f7fe640 1 -- 172.21.15.70:0/1381943225 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c600b54b0 con 0x7f6c6009b650 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c5e7fc640 1 -- 172.21.15.70:0/1381943225 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3512776207 0 0) 0x7f6c600d0350 con 0x7f6c6015c8d0 2024-04-15T15:17:16.081 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.078+0000 7f6c57fff640 1 --2- 172.21.15.70:0/1381943225 >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 0x7f6c600cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.079+0000 7f6c5f7fe640 1 --2- 172.21.15.70:0/1381943225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c600b5f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009d90 tx=0x7f6c5802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0dcb969f2b2d12 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.079+0000 7f6c5e7fc640 1 -- 172.21.15.70:0/1381943225 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58039070 con 0x7f6c6009b650 2024-04-15T15:17:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.079+0000 7f6c5e7fc640 1 -- 172.21.15.70:0/1381943225 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c5802cc50 con 0x7f6c6009b650 2024-04-15T15:17:16.082 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.079+0000 7f6c5e7fc640 1 -- 172.21.15.70:0/1381943225 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c58034440 con 0x7f6c6009b650 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 -- 172.21.15.70:0/1381943225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 msgr2=0x7f6c600b5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 --2- 172.21.15.70:0/1381943225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c600b5f90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6c58009d90 tx=0x7f6c5802bea0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 -- 172.21.15.70:0/1381943225 shutdown_connections 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 --2- 172.21.15.70:0/1381943225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c600b5f90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 --2- 172.21.15.70:0/1381943225 >> v2:172.21.15.96:3300/0 conn(0x7f6c6009b0e0 0x7f6c600cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 -- 172.21.15.70:0/1381943225 >> 172.21.15.70:0/1381943225 conn(0x7f6c600b3fb0 msgr2=0x7f6c600b43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 -- 172.21.15.70:0/1381943225 shutdown_connections 2024-04-15T15:17:16.083 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.080+0000 7f6c654b2640 1 -- 172.21.15.70:0/1381943225 wait complete. 2024-04-15T15:17:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.081+0000 7f6c654b2640 1 Processor -- start 2024-04-15T15:17:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.081+0000 7f6c654b2640 1 -- start start 2024-04-15T15:17:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c6009e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.086 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 0x7f6c600ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6c600b6650 con 0x7f6c6009b0e0 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f6c600b64d0 con 0x7f6c6009ee00 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c654b2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f6c600b67d0 con 0x7f6c6009b650 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49048/0 (socket says 172.21.15.70:49048) 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 -- 172.21.15.70:0/101168746 learned_addr learned my addr 172.21.15.70:0/101168746 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c57fff640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c6009e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5f7fe640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 0x7f6c600ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 -- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 msgr2=0x7f6c6009e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c6009e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 -- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 msgr2=0x7f6c600ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.087 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 0x7f6c600ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 -- 172.21.15.70:0/101168746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c601f5860 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5f7fe640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 0x7f6c600ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.082+0000 7f6c5effd640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000c430 tx=0x7f6c5000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.083+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c500074e0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.083+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c500047e0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.083+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c600cb230 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.083+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c50004bd0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.083+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c600cb7f0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.084+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c600751c0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.084+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f6c50011070 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.085+0000 7f6c577fe640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 0x7f6c4006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.085+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f6c5008b860 con 0x7f6c6009b0e0 2024-04-15T15:17:16.088 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.085+0000 7f6c57fff640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 0x7f6c4006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:16.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.085+0000 7f6c57fff640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 0x7f6c4006ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009560 tx=0x7f6c48009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:16.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.089+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f6c600751c0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.304+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f6c600690f0 con 0x7f6c6009b0e0 2024-04-15T15:17:16.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.962+0000 7f6c577fe640 1 -- 172.21.15.70:0/101168746 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) ==== 104+0+0 (secure 0 0 0) 0x7f6c50059490 con 0x7f6c6009b0e0 2024-04-15T15:17:16.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 msgr2=0x7f6c4006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 0x7f6c4006ee40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6c48009560 tx=0x7f6c48009190 comp rx=0 tx=0).stop 2024-04-15T15:17:16.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 msgr2=0x7f6c601f5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000c430 tx=0x7f6c5000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 shutdown_connections 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f6c6009ee00 0x7f6c600ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f6c6009b650 0x7f6c6009e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f6c4006c980 0x7f6c4006ee40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 --2- 172.21.15.70:0/101168746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6c6009b0e0 0x7f6c601f5320 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 >> 172.21.15.70:0/101168746 conn(0x7f6c600b3fb0 msgr2=0x7f6c600703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 shutdown_connections 2024-04-15T15:17:16.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:16.964+0000 7f6c654b2640 1 -- 172.21.15.70:0/101168746 wait complete. 2024-04-15T15:17:16.980 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-15T15:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 Processor -- start 2024-04-15T15:17:17.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 -- start start 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a7806c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a780d03c0 con 0x7f3a7815e8c0 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a780d2340 con 0x7f3a780d3100 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.090+0000 7f3a7edd4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a7815eca0 con 0x7f3a780d1590 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a7cb70640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f3a7815e8c0 0x7f3a780cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46448/0 (socket says 172.21.15.70:46448) 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a7cb70640 1 -- 172.21.15.70:0/2958844926 learned_addr learned my addr 172.21.15.70:0/2958844926 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a7d371640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a7806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 448129791 0 0) 0x7f3a780d03c0 con 0x7f3a7815e8c0 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a68003520 con 0x7f3a7815e8c0 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3466765824 0 0) 0x7f3a780d2340 con 0x7f3a780d3100 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> v1:172.21.15.96:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a780d03c0 con 0x7f3a780d3100 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1298336256 0 0) 0x7f3a68003520 con 0x7f3a7815e8c0 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 msgr2=0x7f3a7806c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a7806c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 >> v1:172.21.15.96:6789/0 conn(0x7f3a780d3100 legacy=0x7f3a780d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a780d2620 con 0x7f3a7815e8c0 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f3a6c001670 con 0x7f3a7815e8c0 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 >> v1:172.21.15.70:6789/0 conn(0x7f3a7815e8c0 legacy=0x7f3a780cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a680051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7815eca0 con 0x7f3a68004800 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6800ba70 con 0x7f3a68003b70 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a6800bc70 con 0x7f3a780d1590 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a77fff640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a7cb70640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.091+0000 7f3a7d371640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a680051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 -- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 msgr2=0x7f3a680051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780d1590 0x7f3a680051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 -- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a68003b70 msgr2=0x7f3a68003f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a68003b70 0x7f3a68003f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 -- 172.21.15.70:0/2958844926 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a68003520 con 0x7f3a68004800 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a77fff640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3a640018d0 tx=0x7f3a68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6612bc4d17ff2bf server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a64037070 con 0x7f3a68004800 2024-04-15T15:17:17.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a640052d0 con 0x7f3a68004800 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.092+0000 7f3a777fe640 1 -- 172.21.15.70:0/2958844926 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a640014f0 con 0x7f3a68004800 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 -- 172.21.15.70:0/2958844926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 msgr2=0x7f3a68004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/2958844926 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a68004c80 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f3a640018d0 tx=0x7f3a68005830 comp rx=0 tx=0).stop 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7d371640 1 -- 172.21.15.70:0/2958844926 reap_dead start 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 -- 172.21.15.70:0/2958844926 shutdown_connections 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 -- 172.21.15.70:0/2958844926 >> 172.21.15.70:0/2958844926 conn(0x7f3a7808bf60 msgr2=0x7f3a780d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 -- 172.21.15.70:0/2958844926 shutdown_connections 2024-04-15T15:17:17.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.093+0000 7f3a7edd4640 1 -- 172.21.15.70:0/2958844926 wait complete. 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.094+0000 7f3a7edd4640 1 Processor -- start 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.094+0000 7f3a7edd4640 1 -- start start 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a780b71d0 0x7f3a78158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 0x7f3a780c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3a780d2be0 con 0x7f3a68004800 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f3a780d4de0 con 0x7f3a780b71d0 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7edd4640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f3a780d4f60 con 0x7f3a780c80c0 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49068/0 (socket says 172.21.15.70:49068) 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 -- 172.21.15.70:0/695163445 learned_addr learned my addr 172.21.15.70:0/695163445 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:17.101 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a77fff640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a780b71d0 0x7f3a78158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7d371640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 0x7f3a780c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 -- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 msgr2=0x7f3a780c8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 0x7f3a780c8540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 -- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a780b71d0 msgr2=0x7f3a78158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a780b71d0 0x7f3a78158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.095+0000 7f3a7cb70640 1 -- 172.21.15.70:0/695163445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a78160200 con 0x7f3a68004800 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a7d371640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 0x7f3a780c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a7cb70640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c004f20 tx=0x7f3a6c0074f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00f750 con 0x7f3a68004800 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a6c018070 con 0x7f3a68004800 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a781604e0 con 0x7f3a68004800 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c00fb50 con 0x7f3a68004800 2024-04-15T15:17:17.102 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.096+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a78160ee0 con 0x7f3a68004800 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.097+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a781610e0 con 0x7f3a68004800 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.098+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3a6c00fcf0 con 0x7f3a68004800 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.098+0000 7f3a757fa640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 0x7f3a5c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.098+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6302+0+0 (secure 0 0 0) 0x7f3a6c08b8d0 con 0x7f3a68004800 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.098+0000 7f3a77fff640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 0x7f3a5c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:17.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.099+0000 7f3a77fff640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 0x7f3a5c06ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a64034000 tx=0x7f3a64031000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:17.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.103+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3a781610e0 con 0x7f3a68004800 2024-04-15T15:17:17.278 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.276+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f3a780689e0 con 0x7f3a68004800 2024-04-15T15:17:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.963+0000 7f3a757fa640 1 -- 172.21.15.70:0/695163445 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) ==== 181+0+0 (secure 0 0 0) 0x7f3a6c059b50 con 0x7f3a68004800 2024-04-15T15:17:17.965 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_metadata' removed 2024-04-15T15:17:17.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 msgr2=0x7f3a5c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 0x7f3a5c06ee70 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3a64034000 tx=0x7f3a64031000 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 msgr2=0x7f3a780cac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c004f20 tx=0x7f3a6c0074f0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 shutdown_connections 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f3a780c80c0 0x7f3a780c8540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f3a780b71d0 0x7f3a78158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f3a5c06c9b0 0x7f3a5c06ee70 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 --2- 172.21.15.70:0/695163445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3a68004800 0x7f3a780cac90 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.966+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 >> 172.21.15.70:0/695163445 conn(0x7f3a7808bf60 msgr2=0x7f3a780b3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.967+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 shutdown_connections 2024-04-15T15:17:17.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:17.967+0000 7f3a7edd4640 1 -- 172.21.15.70:0/695163445 wait complete. 2024-04-15T15:17:17.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-15T15:17:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.087+0000 7fd0ff637640 1 Processor -- start 2024-04-15T15:17:18.089 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.087+0000 7fd0ff637640 1 -- start start 2024-04-15T15:17:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0ff637640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fd0f80cc6b0 0x7fd0f80b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0ff637640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd0f80d0470 0x7fd0f80d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0ff637640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0f80b38d0 con 0x7fd0f80ce770 2024-04-15T15:17:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0ff637640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f80b36a0 con 0x7fd0f80d0470 2024-04-15T15:17:18.090 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0ff637640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fd0f80b3bb0 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fdbd4640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fd0f80d0470 0x7fd0f80d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fd3d3640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd0f80ce770 0x7fd0f815c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46450/0 (socket says 172.21.15.70:46450) 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fd3d3640 1 -- 172.21.15.70:0/2738004436 learned_addr learned my addr 172.21.15.70:0/2738004436 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fcbd2640 1 --2- 172.21.15.70:0/2738004436 >> v2:172.21.15.96:3300/0 conn(0x7fd0f80cc6b0 0x7fd0f80b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fdbd4640 1 -- 172.21.15.70:0/2738004436 >> v2:172.21.15.96:3300/0 conn(0x7fd0f80cc6b0 msgr2=0x7fd0f80b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fdbd4640 1 --2- 172.21.15.70:0/2738004436 >> v2:172.21.15.96:3300/0 conn(0x7fd0f80cc6b0 0x7fd0f80b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fdbd4640 1 -- 172.21.15.70:0/2738004436 >> v1:172.21.15.70:6789/0 conn(0x7fd0f80ce770 legacy=0x7fd0f815c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0effff640 1 -- 172.21.15.70:0/2738004436 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3285167635 0 0) 0x7fd0f80b38d0 con 0x7fd0f80ce770 2024-04-15T15:17:18.091 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.088+0000 7fd0fdbd4640 1 -- 172.21.15.70:0/2738004436 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f80b3e90 con 0x7fd0f80d0470 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.089+0000 7fd0fdbd4640 1 --2- 172.21.15.70:0/2738004436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80d0470 0x7fd0f80d0900 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80b53a0 tx=0x7fd0f002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28248004480fb9a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.089+0000 7fd0effff640 1 -- 172.21.15.70:0/2738004436 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0039070 con 0x7fd0f80d0470 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.089+0000 7fd0effff640 1 -- 172.21.15.70:0/2738004436 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f002cbf0 con 0x7fd0f80d0470 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.089+0000 7fd0effff640 1 -- 172.21.15.70:0/2738004436 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0034440 con 0x7fd0f80d0470 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 -- 172.21.15.70:0/2738004436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80d0470 msgr2=0x7fd0f80d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 --2- 172.21.15.70:0/2738004436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80d0470 0x7fd0f80d0900 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fd0f80b53a0 tx=0x7fd0f002be40 comp rx=0 tx=0).stop 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 -- 172.21.15.70:0/2738004436 shutdown_connections 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 --2- 172.21.15.70:0/2738004436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80d0470 0x7fd0f80d0900 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 --2- 172.21.15.70:0/2738004436 >> v2:172.21.15.96:3300/0 conn(0x7fd0f80cc6b0 0x7fd0f80b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 -- 172.21.15.70:0/2738004436 >> 172.21.15.70:0/2738004436 conn(0x7fd0f806bbc0 msgr2=0x7fd0f806bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 -- 172.21.15.70:0/2738004436 shutdown_connections 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.090+0000 7fd0ff637640 1 -- 172.21.15.70:0/2738004436 wait complete. 2024-04-15T15:17:18.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.091+0000 7fd0ff637640 1 Processor -- start 2024-04-15T15:17:18.094 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.091+0000 7fd0ff637640 1 -- start start 2024-04-15T15:17:18.096 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 0x7fd0f815a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 0x7fd0f815a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f815c7d0 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fd0f80ceb50 con 0x7fd0f80d0470 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fd0f815c950 con 0x7fd0f80ce770 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49098/0 (socket says 172.21.15.70:49098) 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 -- 172.21.15.70:0/1650114124 learned_addr learned my addr 172.21.15.70:0/1650114124 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fdbd4640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 0x7fd0f815a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fcbd2640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 0x7fd0f815a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 -- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 msgr2=0x7fd0f815a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 0x7fd0f815a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 -- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 msgr2=0x7fd0f815a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 0x7fd0f815a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 -- 172.21.15.70:0/1650114124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0f806f350 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fdbd4640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 0x7fd0f815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fcbd2640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 0x7fd0f815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0fd3d3640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800c4d0 tx=0x7fd0e800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8018070 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e8007500 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e8004a20 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.092+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0f806f630 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.093+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0f806e120 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.094+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0f80b0660 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.094+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fd0e801b070 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.098 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.095+0000 7fd0edffb640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 0x7fd0dc06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:18.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.095+0000 7fd0fcbd2640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 0x7fd0dc06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:18.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.095+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 5864+0+0 (secure 0 0 0) 0x7fd0e808aee0 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.095+0000 7fd0fcbd2640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 0x7fd0dc06ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4006fd0 tx=0x7fd0f4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:18.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.099+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fd0f80b0660 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.277 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.275+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd0f80689e0 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.968+0000 7fd0edffb640 1 -- 172.21.15.70:0/1650114124 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) ==== 169+0+0 (secure 0 0 0) 0x7fd0e8059310 con 0x7fd0f80cc6b0 2024-04-15T15:17:18.970 INFO:teuthology.orchestra.run.smithi070.stderr:pool 'cephfs_data' removed 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 msgr2=0x7fd0dc06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 0x7fd0dc06ee40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fd0f4006fd0 tx=0x7fd0f4008040 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 msgr2=0x7fd0f806e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fd0e800c4d0 tx=0x7fd0e800c990 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 shutdown_connections 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fd0f80d0470 0x7fd0f815a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fd0f80ce770 0x7fd0f815a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fd0dc06c980 0x7fd0dc06ee40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 --2- 172.21.15.70:0/1650114124 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0f80cc6b0 0x7fd0f806e640 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 >> 172.21.15.70:0/1650114124 conn(0x7fd0f806bbc0 msgr2=0x7fd0f80b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 shutdown_connections 2024-04-15T15:17:18.974 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:18.972+0000 7fd0ff637640 1 -- 172.21.15.70:0/1650114124 wait complete. 2024-04-15T15:17:18.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds debug_mds 2024-04-15T15:17:19.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.098+0000 7f019f915640 1 Processor -- start 2024-04-15T15:17:19.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.098+0000 7f019f915640 1 -- start start 2024-04-15T15:17:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019f915640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f019815b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019f915640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f01980ae280 0x7f01980ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019f915640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f019815cb50 con 0x7f01980d0690 2024-04-15T15:17:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019f915640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f019815bfd0 con 0x7f01980ae280 2024-04-15T15:17:19.103 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019f915640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f019815c2b0 con 0x7f01980b3160 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019d6b1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f01980d0690 0x7f01980d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46464/0 (socket says 172.21.15.70:46464) 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019d6b1640 1 -- 172.21.15.70:0/2856085094 learned_addr learned my addr 172.21.15.70:0/2856085094 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3548597873 0 0) 0x7f019815cb50 con 0x7f01980d0690 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f019815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f019deb2640 1 --2- 172.21.15.70:0/2856085094 >> v2:172.21.15.96:3300/0 conn(0x7f01980ae280 0x7f01980ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0184005730 con 0x7f01980d0690 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1264836395 0 0) 0x7f0184005730 con 0x7f01980d0690 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 msgr2=0x7f019815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f019815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 >> v2:172.21.15.96:3300/0 conn(0x7f01980ae280 msgr2=0x7f01980ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 --2- 172.21.15.70:0/2856085094 >> v2:172.21.15.96:3300/0 conn(0x7f01980ae280 0x7f01980ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f019815c590 con 0x7f01980d0690 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f01940019f0 con 0x7f01980d0690 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.099+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 >> v1:172.21.15.70:6789/0 conn(0x7f01980d0690 legacy=0x7f01980d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.104 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f0184005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0184006140 0x7f0184006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0184006f50 0x7f01840073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f019815bfd0 con 0x7f0184006f50 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f019815c2b0 con 0x7f0184006140 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f018400dbd0 con 0x7f01980b3160 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0184006f50 0x7f01840073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019d6b1640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f0184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0184006140 0x7f0184006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 -- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 msgr2=0x7f0184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f01980b3160 0x7f0184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 -- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0184006140 msgr2=0x7f0184006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0184006140 0x7f0184006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019deb2640 1 -- 172.21.15.70:0/2856085094 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0184005730 con 0x7f0184006f50 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.100+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0184006140 0x7f0184006a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:19.105 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019deb2640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0184006f50 0x7f01840073d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ac0 tx=0x7f0184007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f32718054abff7c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190039070 con 0x7f0184006f50 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f019002c6d0 con 0x7f0184006f50 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f018ffff640 1 -- 172.21.15.70:0/2856085094 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0190009690 con 0x7f0184006f50 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 -- 172.21.15.70:0/2856085094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0184006f50 msgr2=0x7f01840073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 --2- 172.21.15.70:0/2856085094 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0184006f50 0x7f01840073d0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f0190009ac0 tx=0x7f0184007a40 comp rx=0 tx=0).stop 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019deb2640 1 -- 172.21.15.70:0/2856085094 reap_dead start 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 -- 172.21.15.70:0/2856085094 shutdown_connections 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 -- 172.21.15.70:0/2856085094 >> 172.21.15.70:0/2856085094 conn(0x7f019806be00 msgr2=0x7f019806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 -- 172.21.15.70:0/2856085094 shutdown_connections 2024-04-15T15:17:19.106 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.101+0000 7f019f915640 1 -- 172.21.15.70:0/2856085094 wait complete. 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 Processor -- start 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 -- start start 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0184006f50 0x7f019814d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0198159870 0x7f0198159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f019815b690 con 0x7f0198156f60 2024-04-15T15:17:19.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f01980b4fb0 con 0x7f0198159870 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019f915640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f019815b810 con 0x7f0184006f50 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019ceb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019ceb0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49118/0 (socket says 172.21.15.70:49118) 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019ceb0640 1 -- 172.21.15.70:0/2926263862 learned_addr learned my addr 172.21.15.70:0/2926263862 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019d6b1640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0184006f50 0x7f019814d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.103+0000 7f019deb2640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0198159870 0x7f0198159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 -- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0184006f50 msgr2=0x7f019814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0184006f50 0x7f019814d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 -- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0198159870 msgr2=0x7f0198159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0198159870 0x7f0198159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 -- 172.21.15.70:0/2926263862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01981551a0 con 0x7f0198156f60 2024-04-15T15:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019ceb0640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0188011570 tx=0x7f0188011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188015870 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0188015a10 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0198155480 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0188015dc0 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.104+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0198155a40 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.105+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0198155e70 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.106+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f018801d400 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.106+0000 7f018dffb640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 0x7f017c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.106+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f018808c1f0 con 0x7f0198156f60 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.106+0000 7f019d6b1640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 0x7f017c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.107+0000 7f019d6b1640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 0x7f017c06ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f019815a9d0 tx=0x7f0194006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.111+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f0188019030 con 0x7f0198156f60 2024-04-15T15:17:19.289 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.287+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "debug_mds"} v 0) -- 0x7f01980689e0 con 0x7f0198156f60 2024-04-15T15:17:19.292 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.290+0000 7f018dffb640 1 -- 172.21.15.70:0/2926263862 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "debug_mds"}]=0 v13) ==== 92+0+0 (secure 0 0 0) 0x7f018805a750 con 0x7f0198156f60 2024-04-15T15:17:19.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 msgr2=0x7f017c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 0x7f017c06ee40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f019815a9d0 tx=0x7f0194006040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 msgr2=0x7f01981563c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0188011570 tx=0x7f0188011970 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 shutdown_connections 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f0198159870 0x7f0198159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f017c06c980 0x7f017c06ee40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0198156f60 0x7f01981563c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.292+0000 7f019f915640 1 --2- 172.21.15.70:0/2926263862 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f0184006f50 0x7f019814d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.293+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 >> 172.21.15.70:0/2926263862 conn(0x7f019806be00 msgr2=0x7f019806e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.293+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 shutdown_connections 2024-04-15T15:17:19.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.293+0000 7f019f915640 1 -- 172.21.15.70:0/2926263862 wait complete. 2024-04-15T15:17:19.305 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_cache_quiesce_splitauth 2024-04-15T15:17:19.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.408+0000 7f9d76fe1640 1 Processor -- start 2024-04-15T15:17:19.410 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.408+0000 7f9d76fe1640 1 -- start start 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d76fe1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d700d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d76fe1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d700d0fb0 con 0x7f9d700b3160 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d76fe1640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7015cb50 con 0x7f9d700ae280 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d76fe1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9d7015bda0 con 0x7f9d700d0690 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d6ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9d700b3160 0x7f9d7015b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46474/0 (socket says 172.21.15.70:46474) 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.409+0000 7f9d6ffff640 1 -- 172.21.15.70:0/3977114075 learned_addr learned my addr 172.21.15.70:0/3977114075 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d700d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 599535289 0 0) 0x7f9d700d0fb0 con 0x7f9d700b3160 2024-04-15T15:17:19.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d58003520 con 0x7f9d700b3160 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3352392890 0 0) 0x7f9d58003520 con 0x7f9d700b3160 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 msgr2=0x7f9d700d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d700d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 >> v1:172.21.15.96:6789/0 conn(0x7f9d700ae280 legacy=0x7f9d700ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7015bfd0 con 0x7f9d700b3160 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f9d64001670 con 0x7f9d700b3160 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 >> v1:172.21.15.70:6789/0 conn(0x7f9d700b3160 legacy=0x7f9d7015b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d58003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d58004800 0x7f9d58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5800ba70 con 0x7f9d580039e0 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9d5800bbf0 con 0x7f9d58004800 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9d5800bdf0 con 0x7f9d700d0690 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d58003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d6ffff640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d58004800 0x7f9d58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d7557e640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 -- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 msgr2=0x7f9d580051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d700d0690 0x7f9d580051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 -- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d58004800 msgr2=0x7f9d58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d58004800 0x7f9d58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.410+0000 7f9d74d7d640 1 -- 172.21.15.70:0/3977114075 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d58003520 con 0x7f9d580039e0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.411+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d58003e00 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009a90 tx=0x7f9d58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bdaf9ec3b490f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.411+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60039070 con 0x7f9d580039e0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.411+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6002c680 con 0x7f9d580039e0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.411+0000 7f9d6f7fe640 1 -- 172.21.15.70:0/3977114075 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60009690 con 0x7f9d580039e0 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 -- 172.21.15.70:0/3977114075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 msgr2=0x7f9d58003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/3977114075 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d58003e00 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9d60009a90 tx=0x7f9d58005830 comp rx=0 tx=0).stop 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d7557e640 1 -- 172.21.15.70:0/3977114075 reap_dead start 2024-04-15T15:17:19.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 -- 172.21.15.70:0/3977114075 shutdown_connections 2024-04-15T15:17:19.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 -- 172.21.15.70:0/3977114075 >> 172.21.15.70:0/3977114075 conn(0x7f9d7006be00 msgr2=0x7f9d7006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 -- 172.21.15.70:0/3977114075 shutdown_connections 2024-04-15T15:17:19.417 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.412+0000 7f9d76fe1640 1 -- 172.21.15.70:0/3977114075 wait complete. 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.413+0000 7f9d76fe1640 1 Processor -- start 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.413+0000 7f9d76fe1640 1 -- start start 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 0x7f9d7009c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d70156360 0x7f9d70156810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7015c870 con 0x7f9d580039e0 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9d7015c590 con 0x7f9d7009a640 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d76fe1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9d700b4fb0 con 0x7f9d70156360 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49140/0 (socket says 172.21.15.70:49140) 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 -- 172.21.15.70:0/4153161 learned_addr learned my addr 172.21.15.70:0/4153161 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d6ffff640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 0x7f9d7009c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d7557e640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d70156360 0x7f9d70156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.420 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 -- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d70156360 msgr2=0x7f9d70156810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d70156360 0x7f9d70156810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 -- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 msgr2=0x7f9d7009c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 0x7f9d7009c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d74d7d640 1 -- 172.21.15.70:0/4153161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7009d1a0 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.414+0000 7f9d6ffff640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 0x7f9d7009c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d74d7d640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9d60037040 tx=0x7f9d60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60042020 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d6003a040 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7009d480 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d60039070 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.415+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d7009da40 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.416+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d700689e0 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.416+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9d60045020 con 0x7f9d580039e0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.417+0000 7f9d6d7fa640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 0x7f9d5c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.417+0000 7f9d6ffff640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 0x7f9d5c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.421 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.417+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f9d600acd30 con 0x7f9d580039e0 2024-04-15T15:17:19.422 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.417+0000 7f9d6ffff640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 0x7f9d5c06ee70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9d64001eb0 tx=0x7f9d64009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.423 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.421+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f9d6004c070 con 0x7f9d580039e0 2024-04-15T15:17:19.597 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.595+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"} v 0) -- 0x7f9d700739e0 con 0x7f9d580039e0 2024-04-15T15:17:19.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.598+0000 7f9d6d7fa640 1 -- 172.21.15.70:0/4153161 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_cache_quiesce_splitauth"}]=0 v14) ==== 110+0+0 (secure 0 0 0) 0x7f9d6007b320 con 0x7f9d580039e0 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.600+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 msgr2=0x7f9d5c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.600+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 0x7f9d5c06ee70 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f9d64001eb0 tx=0x7f9d64009040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.600+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 msgr2=0x7f9d700997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.600+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f9d60037040 tx=0x7f9d60004730 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 shutdown_connections 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9d70156360 0x7f9d70156810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9d7009a640 0x7f9d7009c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9d5c06c9b0 0x7f9d5c06ee70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 --2- 172.21.15.70:0/4153161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d580039e0 0x7f9d700997f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 >> 172.21.15.70:0/4153161 conn(0x7f9d7006be00 msgr2=0x7f9d700b3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 shutdown_connections 2024-04-15T15:17:19.603 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.601+0000 7f9d76fe1640 1 -- 172.21.15.70:0/4153161 wait complete. 2024-04-15T15:17:19.614 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-15T15:17:19.721 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.720+0000 7f45fafa8640 1 Processor -- start 2024-04-15T15:17:19.722 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.720+0000 7f45fafa8640 1 -- start start 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45fafa8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45f4160b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45fafa8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f406f090 con 0x7f45f4067720 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45fafa8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45f415d660 con 0x7f45f415d090 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45fafa8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f45f4160130 con 0x7f45f406ec00 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f45f4067720 0x7f45f40ce7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46488/0 (socket says 172.21.15.70:46488) 2024-04-15T15:17:19.725 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f3fff640 1 -- 172.21.15.70:0/1495315784 learned_addr learned my addr 172.21.15.70:0/1495315784 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f9545640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45f4160b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1725830398 0 0) 0x7f45f406f090 con 0x7f45f4067720 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f45e4003520 con 0x7f45f4067720 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3377315711 0 0) 0x7f45e4003520 con 0x7f45f4067720 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 msgr2=0x7f45f4160b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45f4160b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 >> v1:172.21.15.96:6789/0 conn(0x7f45f415d090 legacy=0x7f45f406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f415d940 con 0x7f45f4067720 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f45e0001670 con 0x7f45f4067720 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 >> v1:172.21.15.70:6789/0 conn(0x7f45f4067720 legacy=0x7f45f40ce7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e40039e0 0x7f45e4003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e4004d40 0x7f45e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.726 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45f4160130 con 0x7f45e4004d40 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45e400baf0 con 0x7f45e40039e0 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f45e400bcf0 con 0x7f45f406ec00 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f9545640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e4004d40 0x7f45e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f3fff640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.721+0000 7f45f8d44640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e40039e0 0x7f45e4003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 -- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 msgr2=0x7f45e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45f406ec00 0x7f45e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 -- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e40039e0 msgr2=0x7f45e4003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45e40039e0 0x7f45e4003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 -- 172.21.15.70:0/1495315784 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e4003520 con 0x7f45e4004d40 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.722+0000 7f45f9545640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e4004d40 0x7f45e40051c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009fc0 tx=0x7f45e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c96dff2786040f server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec039070 con 0x7f45e4004d40 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45ec02cc20 con 0x7f45e4004d40 2024-04-15T15:17:19.727 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45f37fe640 1 -- 172.21.15.70:0/1495315784 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec009c10 con 0x7f45e4004d40 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 -- 172.21.15.70:0/1495315784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e4004d40 msgr2=0x7f45e40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 --2- 172.21.15.70:0/1495315784 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45e4004d40 0x7f45e40051c0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009fc0 tx=0x7f45e4005830 comp rx=0 tx=0).stop 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45f9545640 1 -- 172.21.15.70:0/1495315784 reap_dead start 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 -- 172.21.15.70:0/1495315784 shutdown_connections 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 -- 172.21.15.70:0/1495315784 >> 172.21.15.70:0/1495315784 conn(0x7f45f4073360 msgr2=0x7f45f406b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 -- 172.21.15.70:0/1495315784 shutdown_connections 2024-04-15T15:17:19.728 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.723+0000 7f45fafa8640 1 -- 172.21.15.70:0/1495315784 wait complete. 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.724+0000 7f45fafa8640 1 Processor -- start 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.724+0000 7f45fafa8640 1 -- start start 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e4004d40 0x7f45f41f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 0x7f45f4150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45f40b0460 con 0x7f45f41e5d40 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f45f4160660 con 0x7f45f414ffc0 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45fafa8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f45f40b05e0 con 0x7f45e4004d40 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45f3fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49154/0 (socket says 172.21.15.70:49154) 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45f3fff640 1 -- 172.21.15.70:0/2665946483 learned_addr learned my addr 172.21.15.70:0/2665946483 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.725+0000 7f45f8d44640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e4004d40 0x7f45f41f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f9545640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 0x7f45f4150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 -- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e4004d40 msgr2=0x7f45f41f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e4004d40 0x7f45f41f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 -- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 msgr2=0x7f45f4150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 0x7f45f4150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 -- 172.21.15.70:0/2665946483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f41f5990 con 0x7f45f41e5d40 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f9545640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 0x7f45f4150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f3fff640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f45e0007290 tx=0x7f45e0007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e000f9c0 con 0x7f45f41e5d40 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45e0018070 con 0x7f45f41e5d40 2024-04-15T15:17:19.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f4154fc0 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45e000fda0 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.726+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f414f7c0 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.727+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f414f310 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.727+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f45e001d460 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.728+0000 7f45f17fa640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 0x7f45d806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.728+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f45e008bad0 con 0x7f45f41e5d40 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.728+0000 7f45f8d44640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 0x7f45d806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:19.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.728+0000 7f45f8d44640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 0x7f45d806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f45f4151760 tx=0x7f45e8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:19.734 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.732+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f45f41f5990 con 0x7f45f41e5d40 2024-04-15T15:17:19.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.912+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f45f40689e0 con 0x7f45f41e5d40 2024-04-15T15:17:19.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.916+0000 7f45f17fa640 1 -- 172.21.15.70:0/2665946483 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) ==== 104+0+0 (secure 0 0 0) 0x7f45e005a030 con 0x7f45f41e5d40 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 msgr2=0x7f45d806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 0x7f45d806ee40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f45f4151760 tx=0x7f45e8009040 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 msgr2=0x7f45f41546c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f45e0007290 tx=0x7f45e0007750 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 shutdown_connections 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f45f414ffc0 0x7f45f4150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f45d806c980 0x7f45d806ee40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45f41e5d40 0x7f45f41546c0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 --2- 172.21.15.70:0/2665946483 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f45e4004d40 0x7f45f41f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 >> 172.21.15.70:0/2665946483 conn(0x7f45f4073360 msgr2=0x7f45f40b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:19.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 shutdown_connections 2024-04-15T15:17:19.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:19.918+0000 7f45fafa8640 1 -- 172.21.15.70:0/2665946483 wait complete. 2024-04-15T15:17:19.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked' 2024-04-15T15:17:20.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.033+0000 7fcfe7a0d640 1 Processor -- start 2024-04-15T15:17:20.035 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.033+0000 7fcfe7a0d640 1 -- start start 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe7a0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfe0071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe7a0d640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fcfe00d0da0 0x7fcfe00af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe7a0d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfe015db40 con 0x7fcfe00af3e0 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe7a0d640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fcfe015cfc0 con 0x7fcfe00d0da0 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe7a0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe015d2a0 con 0x7fcfe0072d80 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe4fa8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fcfe00af3e0 0x7fcfe00d0690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46494/0 (socket says 172.21.15.70:46494) 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe4fa8640 1 -- 172.21.15.70:0/2793038225 learned_addr learned my addr 172.21.15.70:0/2793038225 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2793038225 >> v2:172.21.15.96:3300/0 conn(0x7fcfe00d0da0 0x7fcfe00af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfe0071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.034+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 704860660 0 0) 0x7fcfe015db40 con 0x7fcfe00af3e0 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcfcc003520 con 0x7fcfe00af3e0 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 268579436 0 0) 0x7fcfcc003520 con 0x7fcfe00af3e0 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 msgr2=0x7fcfe0071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfe0071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 >> v2:172.21.15.96:3300/0 conn(0x7fcfe00d0da0 msgr2=0x7fcfe00af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 --2- 172.21.15.70:0/2793038225 >> v2:172.21.15.96:3300/0 conn(0x7fcfe00d0da0 0x7fcfe00af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe015d580 con 0x7fcfe00af3e0 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fcfd0001670 con 0x7fcfe00af3e0 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 >> v1:172.21.15.70:6789/0 conn(0x7fcfe00af3e0 legacy=0x7fcfe00d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfcc0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc003f30 0x7fcfcc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc004d40 0x7fcfcc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe015cfc0 con 0x7fcfcc004d40 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe015d2a0 con 0x7fcfcc003f30 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcfcc00ba10 con 0x7fcfe0072d80 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc004d40 0x7fcfcc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe4fa8640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc003f30 0x7fcfcc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfcc0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 msgr2=0x7fcfcc0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfe0072d80 0x7fcfcc0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc003f30 msgr2=0x7fcfcc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfcc003f30 0x7fcfcc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.035+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2793038225 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfcc003520 con 0x7fcfcc004d40 2024-04-15T15:17:20.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.036+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc004d40 0x7fcfcc0051c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e170 tx=0x7fcfcc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d068a7c1cc2d173 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.036+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8039070 con 0x7fcfcc004d40 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.036+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd802cc20 con 0x7fcfcc004d40 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.036+0000 7fcfd7fff640 1 -- 172.21.15.70:0/2793038225 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8009c10 con 0x7fcfcc004d40 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2793038225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc004d40 msgr2=0x7fcfcc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2793038225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfcc004d40 0x7fcfcc0051c0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800e170 tx=0x7fcfcc005830 comp rx=0 tx=0).stop 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2793038225 reap_dead start 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2793038225 shutdown_connections 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2793038225 >> 172.21.15.70:0/2793038225 conn(0x7fcfe006be00 msgr2=0x7fcfe006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2793038225 shutdown_connections 2024-04-15T15:17:20.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.037+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2793038225 wait complete. 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.038+0000 7fcfe7a0d640 1 Processor -- start 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.038+0000 7fcfe7a0d640 1 -- start start 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 0x7fcfe0159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe014d020 0x7fcfe014d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe015cd10 con 0x7fcfe00b3da0 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fcfe015cb90 con 0x7fcfe014d020 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fcfe0071d10 con 0x7fcfcc004d40 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49172/0 (socket says 172.21.15.70:49172) 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2943205324 learned_addr learned my addr 172.21.15.70:0/2943205324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 0x7fcfe0159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe4fa8640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe014d020 0x7fcfe014d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 msgr2=0x7fcfe0159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 0x7fcfe0159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe014d020 msgr2=0x7fcfe014d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe014d020 0x7fcfe014d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 -- 172.21.15.70:0/2943205324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfe00b4b60 con 0x7fcfe00b3da0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 0x7fcfe0159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe5faa640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009f90 tx=0x7fcfd8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8039030 con 0x7fcfe00b3da0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfd803a040 con 0x7fcfe00b3da0 2024-04-15T15:17:20.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfe00cd870 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.039+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8003990 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.040+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfe00cde30 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.041+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfe00689e0 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.041+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fcfd8045030 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.042+0000 7fcfd5ffb640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 0x7fcfc406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.042+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 0x7fcfc406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.042+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fcfd80ada00 con 0x7fcfe00b3da0 2024-04-15T15:17:20.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.042+0000 7fcfe57a9640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 0x7fcfc406ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009bc0 tx=0x7fcfdc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:20.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.046+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fcfd804c020 con 0x7fcfe00b3da0 2024-04-15T15:17:20.291 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.290+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]} v 0) -- 0x7fcfe015bde0 con 0x7fcfe00b3da0 2024-04-15T15:17:20.977 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.975+0000 7fcfd5ffb640 1 -- 172.21.15.70:0/2943205324 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quiesce.TestQuiesce.test_quiesce_blocked"]}]=0 v147) ==== 137+0+0 (secure 0 0 0) 0x7fcfd8034e20 con 0x7fcfe00b3da0 2024-04-15T15:17:20.979 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.977+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 msgr2=0x7fcfc406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.977+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 0x7fcfc406ee40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009bc0 tx=0x7fcfdc008040 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.977+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 msgr2=0x7fcfe00b4200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.977+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8009f90 tx=0x7fcfd8009750 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 shutdown_connections 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fcfc406c980 0x7fcfc406ee40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcfe00b3da0 0x7fcfe00b4200 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fcfe014d020 0x7fcfe014d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 --2- 172.21.15.70:0/2943205324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fcfcc004d40 0x7fcfe0159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 >> 172.21.15.70:0/2943205324 conn(0x7fcfe006be00 msgr2=0x7fcfe006c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 shutdown_connections 2024-04-15T15:17:20.980 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:20.978+0000 7fcfe7a0d640 1 -- 172.21.15.70:0/2943205324 wait complete. 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. ... FAIL 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:20.994 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner:Ran 1 test in 88.833s 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner:FAILED (failures=1) 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:20.995 INFO:tasks.cephfs_test_runner:====================================================================== 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner:FAIL: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner:That a file with ceph.quiesce.block is not quiesced. 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 541, in test_quiesce_blocked 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: self._verify_quiesce(root=self.subvolume) 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 138, in _verify_quiesce 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: return self._verify_quiesce_wrapped(rank, status, root, root_inode, ops, cache, splitauth) 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs/test_quiesce.py", line 272, in _verify_quiesce_wrapped 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: self.assertFalse(lock_type.startswith("i")) 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner:AssertionError: True is not false 2024-04-15T15:17:20.996 INFO:tasks.cephfs_test_runner: 2024-04-15T15:17:20.997 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:21.239 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e5920f1f213b4d82bc133749b3664e3f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_9bf1c4ba967dd24077be6e5b87edda4ca87a6e14/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce) 2024-04-15T15:17:21.241 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-15T15:17:21.249 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-15T15:17:21.346 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-15T15:17:21.346 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:17:21.347 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:21.370 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19879 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:21.370 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-04-15T15:17:21.370 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:21.429 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19879 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248836k,nr_inodes=812209,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497676k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-15T15:17:21.429 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:17:21.429 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:21.449 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056369,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=553 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248840k,nr_inodes=812210,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:21.449 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-04-15T15:17:21.449 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-15T15:17:21.508 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056369,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=553 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248840k,nr_inodes=812210,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497688k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-15T15:17:21.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.508 DEBUG:teuthology.orchestra.run.smithi096:> ip netns list 2024-04-15T15:17:21.567 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-15T15:17:21.567 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-15T15:17:21.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.567 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-15T15:17:21.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.641 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-15T15:17:21.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.714 DEBUG:teuthology.orchestra.run.smithi096:> sudo ip link delete ceph-brx 2024-04-15T15:17:21.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.809 DEBUG:teuthology.orchestra.run.smithi070:> ip netns list 2024-04-15T15:17:21.828 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-15T15:17:21.828 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-15T15:17:21.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.828 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-15T15:17:21.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-15T15:17:21.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-15T15:17:21.979 DEBUG:teuthology.orchestra.run.smithi070:> sudo ip link delete ceph-brx 2024-04-15T15:17:22.088 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-15T15:17:22.098 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-15T15:17:22.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:22.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.200+0000 7f324cc77640 1 Processor -- start 2024-04-15T15:17:22.202 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.200+0000 7f324cc77640 1 -- start start 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f324cc77640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 0x7f32480b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f324cc77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32480d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f324cc77640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32480b47a0 con 0x7f324815eae0 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f324cc77640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f32480b53b0 con 0x7f324815e700 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f324cc77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f32480b4a80 con 0x7f32480d2360 2024-04-15T15:17:22.204 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f3246d9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f324815eae0 0x7f32480cf370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:46510/0 (socket says 172.21.15.70:46510) 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f3246d9d640 1 -- 172.21.15.70:0/3530182960 learned_addr learned my addr 172.21.15.70:0/3530182960 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.201+0000 7f3245d9b640 1 --2- 172.21.15.70:0/3530182960 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32480d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f324659c640 1 --2- 172.21.15.70:0/3530182960 >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 0x7f32480b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3205169762 0 0) 0x7f32480b47a0 con 0x7f324815eae0 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3234003520 con 0x7f324815eae0 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f3245d9b640 1 -- 172.21.15.70:0/3530182960 >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 msgr2=0x7f32480b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f3245d9b640 1 --2- 172.21.15.70:0/3530182960 >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 0x7f32480b5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2549136483 0 0) 0x7f3234003520 con 0x7f324815eae0 2024-04-15T15:17:22.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f3245d9b640 1 -- 172.21.15.70:0/3530182960 >> v1:172.21.15.70:6789/0 conn(0x7f324815eae0 legacy=0x7f32480cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f3245d9b640 1 -- 172.21.15.70:0/3530182960 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32480b4d60 con 0x7f32480d2360 2024-04-15T15:17:22.206 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.202+0000 7f324659c640 1 --2- 172.21.15.70:0/3530182960 >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 0x7f32480b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f3245d9b640 1 --2- 172.21.15.70:0/3530182960 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32480d2b60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32300097a0 tx=0x7f323002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f8f8f8115ef21f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230037070 con 0x7f32480d2360 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3230030510 con 0x7f32480d2360 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f324559a640 1 -- 172.21.15.70:0/3530182960 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240001670 con 0x7f32480d2360 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f324cc77640 1 -- 172.21.15.70:0/3530182960 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 msgr2=0x7f32480d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.203+0000 7f324cc77640 1 --2- 172.21.15.70:0/3530182960 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32480d2b60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32300097a0 tx=0x7f323002b790 comp rx=0 tx=0).stop 2024-04-15T15:17:22.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 -- 172.21.15.70:0/3530182960 shutdown_connections 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 --2- 172.21.15.70:0/3530182960 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32480d2b60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 --2- 172.21.15.70:0/3530182960 >> v2:172.21.15.96:3300/0 conn(0x7f324815e700 0x7f32480b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 -- 172.21.15.70:0/3530182960 >> 172.21.15.70:0/3530182960 conn(0x7f324808bf60 msgr2=0x7f32480d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 -- 172.21.15.70:0/3530182960 shutdown_connections 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.204+0000 7f324cc77640 1 -- 172.21.15.70:0/3530182960 wait complete. 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.205+0000 7f324cc77640 1 Processor -- start 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.205+0000 7f324cc77640 1 -- start start 2024-04-15T15:17:22.210 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32481f88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.211 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 0x7f3248153640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f32480b5d80 con 0x7f324815eae0 2024-04-15T15:17:22.212 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f32480b4450 con 0x7f324815e700 2024-04-15T15:17:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f32480d1d40 con 0x7f32480d2360 2024-04-15T15:17:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:49176/0 (socket says 172.21.15.70:49176) 2024-04-15T15:17:22.213 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 -- 172.21.15.70:0/734789186 learned_addr learned my addr 172.21.15.70:0/734789186 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:22.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324659c640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32481f88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3245d9b640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 0x7f3248153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 -- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 msgr2=0x7f32481f88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.214 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32481f88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 -- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 msgr2=0x7f3248153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 0x7f3248153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 -- 172.21.15.70:0/734789186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32481f8de0 con 0x7f324815eae0 2024-04-15T15:17:22.215 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324659c640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32481f88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:22.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3245d9b640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 0x7f3248153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:22.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f3246d9d640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f324000c5b0 tx=0x7f324000ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.216 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240005ce0 con 0x7f324815eae0 2024-04-15T15:17:22.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3240007070 con 0x7f324815eae0 2024-04-15T15:17:22.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3248161d00 con 0x7f324815eae0 2024-04-15T15:17:22.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3240007460 con 0x7f324815eae0 2024-04-15T15:17:22.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.206+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32481f51c0 con 0x7f324815eae0 2024-04-15T15:17:22.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.207+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32480689e0 con 0x7f324815eae0 2024-04-15T15:17:22.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.208+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3240007600 con 0x7f324815eae0 2024-04-15T15:17:22.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.208+0000 7f323effd640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 0x7f322806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.208+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f324008b790 con 0x7f324815eae0 2024-04-15T15:17:22.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.208+0000 7f324659c640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 0x7f322806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.209+0000 7f324659c640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 0x7f322806ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3238009bd0 tx=0x7f323800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.221 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.213+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3240059730 con 0x7f324815eae0 2024-04-15T15:17:22.386 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.385+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f32480ba500 con 0x7f322806c980 2024-04-15T15:17:22.387 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.385+0000 7f323effd640 1 -- 172.21.15.70:0/734789186 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13432 (secure 0 0 0) 0x7f32480ba500 con 0x7f322806c980 2024-04-15T15:17:22.387 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:22.388 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:17:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 msgr2=0x7f322806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 0x7f322806ee40 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3238009bd0 tx=0x7f323800b040 comp rx=0 tx=0).stop 2024-04-15T15:17:22.389 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 msgr2=0x7f324806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f324000c5b0 tx=0x7f324000ca70 comp rx=0 tx=0).stop 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 shutdown_connections 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f322806c980 0x7f322806ee40 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f324815eae0 0x7f324806dcd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f324815e700 0x7f3248153640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 --2- 172.21.15.70:0/734789186 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f32480d2360 0x7f32481f88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 >> 172.21.15.70:0/734789186 conn(0x7f324808bf60 msgr2=0x7f32480d0e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.390 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 shutdown_connections 2024-04-15T15:17:22.391 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.387+0000 7f324cc77640 1 -- 172.21.15.70:0/734789186 wait complete. 2024-04-15T15:17:22.400 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-15T15:17:22.095919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238004,"kb_used_data":5212,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736716,"statfs":{"total":773094113280,"available":772850397184,"internally_reserved":0,"allocated":5337088,"data_stored":3046754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-92,"num_write_kb":-96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.135909"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:17.973299+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:17:17.973299+0000","last_peered":"2024-04-15T15:17:17.973299+0000","last_clean":"2024-04-15T15:17:17.973299+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:17:17.973299+0000","last_undegraded":"2024-04-15T15:17:17.973299+0000","last_fullsized":"2024-04-15T15:17:17.973299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542176,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343676,"statfs":{"total":96636764160,"available":96607924224,"internally_reserved":0,"allocated":1171456,"data_stored":871913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":495616,"data_stored":213730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":561152,"data_stored":274788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":804020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:22.401 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.503+0000 7f20d24c7640 1 Processor -- start 2024-04-15T15:17:22.505 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.503+0000 7f20d24c7640 1 -- start start 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d24c7640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20cc0b3160 0x7f20cc15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d24c7640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f20cc0ae280 0x7f20cc0ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d24c7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20cc15cb50 con 0x7f20cc0d0690 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d24c7640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f20cc15bfd0 con 0x7f20cc0b3160 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d24c7640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f20cc15c2b0 con 0x7f20cc0ae280 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20c3fff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f20cc0b3160 0x7f20cc15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.507 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20cbfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f20cc0d0690 0x7f20cc0d0a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49752/0 (socket says 172.21.15.70:49752) 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20cbfff640 1 -- 172.21.15.70:0/1110208904 learned_addr learned my addr 172.21.15.70:0/1110208904 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20c3fff640 1 -- 172.21.15.70:0/1110208904 >> v2:172.21.15.96:3300/0 conn(0x7f20cc0ae280 msgr2=0x7f20cc0ae710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20d0a64640 1 --2- 172.21.15.70:0/1110208904 >> v2:172.21.15.96:3300/0 conn(0x7f20cc0ae280 0x7f20cc0ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20cb7fe640 1 -- 172.21.15.70:0/1110208904 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1792427169 0 0) 0x7f20cc15cb50 con 0x7f20cc0d0690 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20c3fff640 1 --2- 172.21.15.70:0/1110208904 >> v2:172.21.15.96:3300/0 conn(0x7f20cc0ae280 0x7f20cc0ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20c3fff640 1 -- 172.21.15.70:0/1110208904 >> v1:172.21.15.70:6789/0 conn(0x7f20cc0d0690 legacy=0x7f20cc0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.504+0000 7f20c3fff640 1 -- 172.21.15.70:0/1110208904 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc15c590 con 0x7f20cc0b3160 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.505+0000 7f20c3fff640 1 --2- 172.21.15.70:0/1110208904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0b3160 0x7f20cc15b690 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0b1df0 tx=0x7f20b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01da86495c4f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.508 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.505+0000 7f20cb7fe640 1 -- 172.21.15.70:0/1110208904 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4035020 con 0x7f20cc0b3160 2024-04-15T15:17:22.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.505+0000 7f20cb7fe640 1 -- 172.21.15.70:0/1110208904 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b4031830 con 0x7f20cc0b3160 2024-04-15T15:17:22.509 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.505+0000 7f20cb7fe640 1 -- 172.21.15.70:0/1110208904 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4031c00 con 0x7f20cc0b3160 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.505+0000 7f20d24c7640 1 -- 172.21.15.70:0/1110208904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0b3160 msgr2=0x7f20cc15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 --2- 172.21.15.70:0/1110208904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0b3160 0x7f20cc15b690 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f20cc0b1df0 tx=0x7f20b402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 -- 172.21.15.70:0/1110208904 shutdown_connections 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 --2- 172.21.15.70:0/1110208904 >> v2:172.21.15.96:3300/0 conn(0x7f20cc0ae280 0x7f20cc0ae710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 --2- 172.21.15.70:0/1110208904 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0b3160 0x7f20cc15b690 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.513 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 -- 172.21.15.70:0/1110208904 >> 172.21.15.70:0/1110208904 conn(0x7f20cc06be00 msgr2=0x7f20cc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 -- 172.21.15.70:0/1110208904 shutdown_connections 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.506+0000 7f20d24c7640 1 -- 172.21.15.70:0/1110208904 wait complete. 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 Processor -- start 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 -- start start 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20cc0b3160 0x7f20cc159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20cc0d0690 0x7f20cc09a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.514 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc15bbd0 con 0x7f20cc0ae280 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f20cc0b4fb0 con 0x7f20cc0b3160 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.507+0000 7f20d24c7640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f20cc15bd50 con 0x7f20cc0d0690 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51510/0 (socket says 172.21.15.70:51510) 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 -- 172.21.15.70:0/2264868321 learned_addr learned my addr 172.21.15.70:0/2264868321 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:22.515 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 -- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20cc0d0690 msgr2=0x7f20cc09a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20cc0d0690 0x7f20cc09a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 -- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20cc0b3160 msgr2=0x7f20cc159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20cc0b3160 0x7f20cc159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 -- 172.21.15.70:0/2264868321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20cc09d530 con 0x7f20cc0ae280 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20cbfff640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00c430 tx=0x7f20bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.516 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc0074e0 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20bc0047e0 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20cc09d810 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20bc004bd0 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.508+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20cc09ddd0 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.509+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20cc072f20 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.510+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f20bc011070 con 0x7f20cc0ae280 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.510+0000 7f20c97fa640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 0x7f20ac06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.517 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.510+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f20bc08b6d0 con 0x7f20cc0ae280 2024-04-15T15:17:22.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.510+0000 7f20c3fff640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 0x7f20ac06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.511+0000 7f20c3fff640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 0x7f20ac06ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.518 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.514+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f20bc059cc0 con 0x7f20cc0ae280 2024-04-15T15:17:22.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.686+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f20cc0689e0 con 0x7f20ac06c980 2024-04-15T15:17:22.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.687+0000 7f20c97fa640 1 -- 172.21.15.70:0/2264868321 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13432 (secure 0 0 0) 0x7f20cc0689e0 con 0x7f20ac06c980 2024-04-15T15:17:22.689 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:22.690 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:17:22.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 msgr2=0x7f20ac06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 0x7f20ac06ee40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f20b4036000 tx=0x7f20b4030000 comp rx=0 tx=0).stop 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 msgr2=0x7f20cc09cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00c430 tx=0x7f20bc00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 shutdown_connections 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f20cc0d0690 0x7f20cc09a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f20cc0b3160 0x7f20cc159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f20ac06c980 0x7f20ac06ee40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 --2- 172.21.15.70:0/2264868321 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f20cc0ae280 0x7f20cc09cec0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 >> 172.21.15.70:0/2264868321 conn(0x7f20cc06be00 msgr2=0x7f20cc071d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 shutdown_connections 2024-04-15T15:17:22.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.689+0000 7f20d24c7640 1 -- 172.21.15.70:0/2264868321 wait complete. 2024-04-15T15:17:22.702 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-15T15:17:22.095919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238004,"kb_used_data":5212,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736716,"statfs":{"total":773094113280,"available":772850397184,"internally_reserved":0,"allocated":5337088,"data_stored":3046754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-92,"num_write_kb":-96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.135909"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:17.973299+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:17:17.973299+0000","last_peered":"2024-04-15T15:17:17.973299+0000","last_clean":"2024-04-15T15:17:17.973299+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:17:17.973299+0000","last_undegraded":"2024-04-15T15:17:17.973299+0000","last_fullsized":"2024-04-15T15:17:17.973299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542176,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343676,"statfs":{"total":96636764160,"available":96607924224,"internally_reserved":0,"allocated":1171456,"data_stored":871913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":495616,"data_stored":213730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":561152,"data_stored":274788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":804020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:22.703 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-15T15:17:22.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:22.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.805+0000 7fa244a60640 1 Processor -- start 2024-04-15T15:17:22.807 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.805+0000 7fa244a60640 1 -- start start 2024-04-15T15:17:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.805+0000 7fa244a60640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fa24015e830 0x7fa2400b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.810 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa244a60640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa240067e30 con 0x7fa2400b53c0 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa244a60640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2400b3c10 con 0x7fa24015ec10 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa244a60640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fa2400b3ef0 con 0x7fa24015e830 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23ed9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa2400b53c0 0x7fa2400aeac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49766/0 (socket says 172.21.15.70:49766) 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23ed9d640 1 -- 172.21.15.70:0/2811798941 learned_addr learned my addr 172.21.15.70:0/2811798941 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23e59c640 1 --2- 172.21.15.70:0/2811798941 >> v2:172.21.15.96:3300/0 conn(0x7fa24015e830 0x7fa2400b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 980203744 0 0) 0x7fa240067e30 con 0x7fa2400b53c0 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa22c003520 con 0x7fa2400b53c0 2024-04-15T15:17:22.811 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2187737033 0 0) 0x7fa22c003520 con 0x7fa2400b53c0 2024-04-15T15:17:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 >> v2:172.21.15.96:3300/0 conn(0x7fa24015e830 msgr2=0x7fa2400b4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 --2- 172.21.15.70:0/2811798941 >> v2:172.21.15.96:3300/0 conn(0x7fa24015e830 0x7fa2400b4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 >> v1:172.21.15.96:6789/0 conn(0x7fa24015ec10 legacy=0x7fa240067720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2400b41d0 con 0x7fa2400b53c0 2024-04-15T15:17:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fa2380019f0 con 0x7fa2400b53c0 2024-04-15T15:17:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 >> v1:172.21.15.70:6789/0 conn(0x7fa2400b53c0 legacy=0x7fa2400aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 0x7fa22c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c004800 0x7fa22c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa22c0051c0 0x7fa22c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2400b3ef0 con 0x7fa22c0039b0 2024-04-15T15:17:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c00c160 con 0x7fa22c0051c0 2024-04-15T15:17:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa22c00c360 con 0x7fa22c004800 2024-04-15T15:17:22.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa23e59c640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 0x7fa22c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.806+0000 7fa235d9b640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c004800 0x7fa22c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23ed9d640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa22c0051c0 0x7fa22c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 -- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c004800 msgr2=0x7fa22c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:22.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c004800 0x7fa22c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 -- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa22c0051c0 msgr2=0x7fa22c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa22c0051c0 0x7fa22c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 -- 172.21.15.70:0/2811798941 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22c003520 con 0x7fa22c0039b0 2024-04-15T15:17:22.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa235d9b640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c004800 0x7fa22c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:22.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23ed9d640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa22c0051c0 0x7fa22c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:22.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23ed9d640 1 -- 172.21.15.70:0/2811798941 reap_dead start 2024-04-15T15:17:22.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.807+0000 7fa23e59c640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 0x7fa22c003dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa230009ac0 tx=0x7fa22c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c277c8f5828d51b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230039070 con 0x7fa22c0039b0 2024-04-15T15:17:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23002c6b0 con 0x7fa22c0039b0 2024-04-15T15:17:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa23dd9b640 1 -- 172.21.15.70:0/2811798941 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa230009690 con 0x7fa22c0039b0 2024-04-15T15:17:22.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 -- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 msgr2=0x7fa22c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 0x7fa22c003dd0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fa230009ac0 tx=0x7fa22c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 -- 172.21.15.70:0/2811798941 shutdown_connections 2024-04-15T15:17:22.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 --2- 172.21.15.70:0/2811798941 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa22c0039b0 0x7fa22c003dd0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 -- 172.21.15.70:0/2811798941 >> 172.21.15.70:0/2811798941 conn(0x7fa240073360 msgr2=0x7fa24006b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 -- 172.21.15.70:0/2811798941 shutdown_connections 2024-04-15T15:17:22.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.808+0000 7fa244a60640 1 -- 172.21.15.70:0/2811798941 wait complete. 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 Processor -- start 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 -- start start 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c0039b0 0x7fa240148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2401f5360 0x7fa24006e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.823 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa24015ff60 con 0x7fa24006eed0 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2400b4790 con 0x7fa2401f5360 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa244a60640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa2401600e0 con 0x7fa22c0039b0 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.810+0000 7fa23ed9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51526/0 (socket says 172.21.15.70:51526) 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 -- 172.21.15.70:0/3066345500 learned_addr learned my addr 172.21.15.70:0/3066345500 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa235d9b640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2401f5360 0x7fa24006e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23e59c640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c0039b0 0x7fa240148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.824 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 -- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c0039b0 msgr2=0x7fa240148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c0039b0 0x7fa240148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 -- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2401f5360 msgr2=0x7fa24006e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2401f5360 0x7fa24006e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 -- 172.21.15.70:0/3066345500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2401e58e0 con 0x7fa24006eed0 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa23ed9d640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b0780 tx=0x7fa23800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa23800eb90 con 0x7fa24006eed0 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa23800ed30 con 0x7fa24006eed0 2024-04-15T15:17:22.825 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24015bb00 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238014670 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.811+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2401e5600 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.812+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2401e5110 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.813+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa238014810 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.813+0000 7fa2377fe640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 0x7fa22006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.813+0000 7fa23e59c640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 0x7fa22006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.813+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fa23808b7e0 con 0x7fa24006eed0 2024-04-15T15:17:22.826 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.814+0000 7fa23e59c640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 0x7fa22006ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:22.827 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.817+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fa238059dd0 con 0x7fa24006eed0 2024-04-15T15:17:22.992 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.990+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa2400689e0 con 0x7fa22006c980 2024-04-15T15:17:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.991+0000 7fa2377fe640 1 -- 172.21.15.70:0/3066345500 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13432 (secure 0 0 0) 0x7fa230047490 con 0x7fa22006c980 2024-04-15T15:17:22.993 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:22.993 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:17:22.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 msgr2=0x7fa22006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 0x7fa22006ee40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa230036000 tx=0x7fa230030000 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 msgr2=0x7fa240158cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fa2400b0780 tx=0x7fa23800e530 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 shutdown_connections 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa22006c980 0x7fa22006ee40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa24006eed0 0x7fa240158cd0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa2401f5360 0x7fa24006e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 --2- 172.21.15.70:0/3066345500 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa22c0039b0 0x7fa240148620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:22.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 >> 172.21.15.70:0/3066345500 conn(0x7fa240073360 msgr2=0x7fa2400b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.993+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 shutdown_connections 2024-04-15T15:17:22.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:22.994+0000 7fa244a60640 1 -- 172.21.15.70:0/3066345500 wait complete. 2024-04-15T15:17:23.006 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-04-15T15:17:22.095919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":238004,"kb_used_data":5212,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736716,"statfs":{"total":773094113280,"available":772850397184,"internally_reserved":0,"allocated":5337088,"data_stored":3046754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62067,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-129,"num_write":-92,"num_write_kb":-96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.135909"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":214,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-04-15T15:17:17.973299+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:17:17.973299+0000","last_peered":"2024-04-15T15:17:17.973299+0000","last_clean":"2024-04-15T15:17:17.973299+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:17:17.973299+0000","last_undegraded":"2024-04-15T15:17:17.973299+0000","last_fullsized":"2024-04-15T15:17:17.973299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542176,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28164,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343676,"statfs":{"total":96636764160,"available":96607924224,"internally_reserved":0,"allocated":1171456,"data_stored":871913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574879,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574878,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27500,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344340,"statfs":{"total":96636764160,"available":96608604160,"internally_reserved":0,"allocated":495616,"data_stored":213730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":507904,"data_stored":220810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574879,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":548,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":561152,"data_stored":274788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574879,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574880,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":220296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574879,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1085440,"data_stored":804020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:23.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-15T15:17:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.109+0000 7fb182dfa640 1 Processor -- start 2024-04-15T15:17:23.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.109+0000 7fb182dfa640 1 -- start start 2024-04-15T15:17:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb182dfa640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fb17c15e700 0x7fb17c0b5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb182dfa640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb17c0d2360 0x7fb17c0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb182dfa640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb17c0b47a0 con 0x7fb17c15eae0 2024-04-15T15:17:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb182dfa640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fb17c0b53b0 con 0x7fb17c0d2360 2024-04-15T15:17:23.113 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb182dfa640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fb17c0b4a80 con 0x7fb17c15e700 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb17bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb17c0d2360 0x7fb17c0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb17bfff640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fb17c0d2360 0x7fb17c0d2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51532/0 (socket says 172.21.15.70:51532) 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.110+0000 7fb17bfff640 1 -- 172.21.15.70:0/224686551 learned_addr learned my addr 172.21.15.70:0/224686551 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17bfff640 1 -- 172.21.15.70:0/224686551 >> v2:172.21.15.96:3300/0 conn(0x7fb17c15e700 msgr2=0x7fb17c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17bfff640 1 --2- 172.21.15.70:0/224686551 >> v2:172.21.15.96:3300/0 conn(0x7fb17c15e700 0x7fb17c0b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17bfff640 1 -- 172.21.15.70:0/224686551 >> v1:172.21.15.96:6789/0 conn(0x7fb17c15eae0 legacy=0x7fb17c0cf370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17bfff640 1 -- 172.21.15.70:0/224686551 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c0b4d60 con 0x7fb17c0d2360 2024-04-15T15:17:23.114 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17bfff640 1 --2- 172.21.15.70:0/224686551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c0d2360 0x7fb17c0d2b60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb17c06c6c0 tx=0x7fb17002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81218e86ffa49806 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17b7fe640 1 -- 172.21.15.70:0/224686551 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb17002ca80 con 0x7fb17c0d2360 2024-04-15T15:17:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17b7fe640 1 -- 172.21.15.70:0/224686551 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb17002cc20 con 0x7fb17c0d2360 2024-04-15T15:17:23.115 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.111+0000 7fb17b7fe640 1 -- 172.21.15.70:0/224686551 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1700343f0 con 0x7fb17c0d2360 2024-04-15T15:17:23.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 -- 172.21.15.70:0/224686551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c0d2360 msgr2=0x7fb17c0d2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 --2- 172.21.15.70:0/224686551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c0d2360 0x7fb17c0d2b60 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fb17c06c6c0 tx=0x7fb17002bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 -- 172.21.15.70:0/224686551 shutdown_connections 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 --2- 172.21.15.70:0/224686551 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c0d2360 0x7fb17c0d2b60 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 --2- 172.21.15.70:0/224686551 >> v2:172.21.15.96:3300/0 conn(0x7fb17c15e700 0x7fb17c0b5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 -- 172.21.15.70:0/224686551 >> 172.21.15.70:0/224686551 conn(0x7fb17c08bf60 msgr2=0x7fb17c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 -- 172.21.15.70:0/224686551 shutdown_connections 2024-04-15T15:17:23.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.112+0000 7fb182dfa640 1 -- 172.21.15.70:0/224686551 wait complete. 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.113+0000 7fb182dfa640 1 Processor -- start 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.113+0000 7fb182dfa640 1 -- start start 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 0x7fb17c0caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb17c15e700 0x7fb17c0b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c0b5d80 con 0x7fb17c15eae0 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fb17c0b4450 con 0x7fb17c15e700 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb182dfa640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fb17c0d1d40 con 0x7fb17c0d2360 2024-04-15T15:17:23.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51534/0 (socket says 172.21.15.70:51534) 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 -- 172.21.15.70:0/4210781355 learned_addr learned my addr 172.21.15.70:0/4210781355 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb17bfff640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb17c15e700 0x7fb17c0b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb180b96640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 0x7fb17c0caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 -- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 msgr2=0x7fb17c0caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 0x7fb17c0caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 -- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb17c15e700 msgr2=0x7fb17c0b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb17c15e700 0x7fb17c0b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb181397640 1 -- 172.21.15.70:0/4210781355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb17c0b9920 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.114+0000 7fb180b96640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 0x7fb17c0caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb181397640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb17400c3c0 tx=0x7fb17400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb174017070 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1740035f0 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1740039e0 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb17c0b9c00 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.115+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb17c0ba1c0 con 0x7fb17c15eae0 2024-04-15T15:17:23.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.116+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb17c0c8120 con 0x7fb17c15eae0 2024-04-15T15:17:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.116+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fb17401a070 con 0x7fb17c15eae0 2024-04-15T15:17:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.117+0000 7fb1797fa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 0x7fb16806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:23.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.117+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fb17408ae30 con 0x7fb17c15eae0 2024-04-15T15:17:23.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.117+0000 7fb180b96640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 0x7fb16806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:23.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.117+0000 7fb180b96640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 0x7fb16806ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00a8f0 tx=0x7fb16c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:23.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.121+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fb174059390 con 0x7fb17c15eae0 2024-04-15T15:17:23.301 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.299+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb17c0689e0 con 0x7fb17c15eae0 2024-04-15T15:17:23.302 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.300+0000 7fb1797fa640 1 -- 172.21.15.70:0/4210781355 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) ==== 74+0+13363 (secure 0 0 0) 0x7fb174058c90 con 0x7fb17c15eae0 2024-04-15T15:17:23.302 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:23.303 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":45,"fsid":"074678c8-a3ad-4fb8-89f1-2c7d457fbbe5","created":"2024-04-15T15:14:45.828531+0000","modified":"2024-04-15T15:17:18.964943+0000","last_up_change":"2024-04-15T15:14:57.226862+0000","last_in_change":"2024-04-15T15:14:48.092043+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T15:14:58.086322+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"03f8cc03-b81e-497c-91dd-ae79aba85fa5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6817","nonce":2850943089}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6819","nonce":2850943089}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6823","nonce":2850943089}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":2850943089},{"type":"v1","addr":"172.21.15.70:6821","nonce":2850943089}]},"public_addr":"172.21.15.70:6817/2850943089","cluster_addr":"172.21.15.70:6819/2850943089","heartbeat_back_addr":"172.21.15.70:6823/2850943089","heartbeat_front_addr":"172.21.15.70:6821/2850943089","state":["exists","up"]},{"osd":1,"uuid":"38bc545f-bf09-4eb5-9ec9-4f66db112bfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6801","nonce":1494795690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6803","nonce":1494795690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6807","nonce":1494795690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1494795690},{"type":"v1","addr":"172.21.15.70:6805","nonce":1494795690}]},"public_addr":"172.21.15.70:6801/1494795690","cluster_addr":"172.21.15.70:6803/1494795690","heartbeat_back_addr":"172.21.15.70:6807/1494795690","heartbeat_front_addr":"172.21.15.70:6805/1494795690","state":["exists","up"]},{"osd":2,"uuid":"3be06c65-ca7d-4627-96ce-ecfd69b5d1c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6809","nonce":3050098076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6811","nonce":3050098076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6815","nonce":3050098076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3050098076},{"type":"v1","addr":"172.21.15.70:6813","nonce":3050098076}]},"public_addr":"172.21.15.70:6809/3050098076","cluster_addr":"172.21.15.70:6811/3050098076","heartbeat_back_addr":"172.21.15.70:6815/3050098076","heartbeat_front_addr":"172.21.15.70:6813/3050098076","state":["exists","up"]},{"osd":3,"uuid":"8710d76b-5689-4f90-ae4a-b9d135325ead","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6825","nonce":1402304147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6827","nonce":1402304147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6831","nonce":1402304147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":1402304147},{"type":"v1","addr":"172.21.15.70:6829","nonce":1402304147}]},"public_addr":"172.21.15.70:6825/1402304147","cluster_addr":"172.21.15.70:6827/1402304147","heartbeat_back_addr":"172.21.15.70:6831/1402304147","heartbeat_front_addr":"172.21.15.70:6829/1402304147","state":["exists","up"]},{"osd":4,"uuid":"2a0965cf-b047-4bae-8af5-03d83051fb3d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6801","nonce":290497787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6803","nonce":290497787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6807","nonce":290497787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":290497787},{"type":"v1","addr":"172.21.15.96:6805","nonce":290497787}]},"public_addr":"172.21.15.96:6801/290497787","cluster_addr":"172.21.15.96:6803/290497787","heartbeat_back_addr":"172.21.15.96:6807/290497787","heartbeat_front_addr":"172.21.15.96:6805/290497787","state":["exists","up"]},{"osd":5,"uuid":"df277b1e-2fe5-48a8-907b-364d770dbe65","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6809","nonce":660710685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6811","nonce":660710685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6815","nonce":660710685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":660710685},{"type":"v1","addr":"172.21.15.96:6813","nonce":660710685}]},"public_addr":"172.21.15.96:6809/660710685","cluster_addr":"172.21.15.96:6811/660710685","heartbeat_back_addr":"172.21.15.96:6815/660710685","heartbeat_front_addr":"172.21.15.96:6813/660710685","state":["exists","up"]},{"osd":6,"uuid":"bfc1e2f6-658b-4c4b-9871-16b54fb2b075","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6817","nonce":2431924296}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6819","nonce":2431924296}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6823","nonce":2431924296}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":2431924296},{"type":"v1","addr":"172.21.15.96:6821","nonce":2431924296}]},"public_addr":"172.21.15.96:6817/2431924296","cluster_addr":"172.21.15.96:6819/2431924296","heartbeat_back_addr":"172.21.15.96:6823/2431924296","heartbeat_front_addr":"172.21.15.96:6821/2431924296","state":["exists","up"]},{"osd":7,"uuid":"2adb9d7e-7c61-48e4-8c49-c3b4929d2954","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6825","nonce":1112579369}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6826","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6827","nonce":1112579369}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6830","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6831","nonce":1112579369}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6828","nonce":1112579369},{"type":"v1","addr":"172.21.15.96:6829","nonce":1112579369}]},"public_addr":"172.21.15.96:6825/1112579369","cluster_addr":"172.21.15.96:6827/1112579369","heartbeat_back_addr":"172.21.15.96:6831/1112579369","heartbeat_front_addr":"172.21.15.96:6829/1112579369","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.254967+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.160405+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242850+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.301005+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.212449+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.219172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.231827+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T15:14:53.242211+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.96:6835/1131645259":"2024-04-16T15:17:15.957710+0000","172.21.15.96:6833/1131645259":"2024-04-16T15:17:15.957710+0000","172.21.15.96:6835/2334246723":"2024-04-16T15:17:00.083643+0000","172.21.15.96:6833/2334246723":"2024-04-16T15:17:00.083643+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T15:17:23.304 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 msgr2=0x7fb16806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 0x7fb16806ee40 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb16c00a8f0 tx=0x7fb16c008040 comp rx=0 tx=0).stop 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 msgr2=0x7fb17c160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fb17400c3c0 tx=0x7fb17400c880 comp rx=0 tx=0).stop 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 shutdown_connections 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fb16806c980 0x7fb16806ee40 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb17c15eae0 0x7fb17c160020 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.305 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fb17c15e700 0x7fb17c0b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 --2- 172.21.15.70:0/4210781355 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fb17c0d2360 0x7fb17c0caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:23.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.302+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 >> 172.21.15.70:0/4210781355 conn(0x7fb17c08bf60 msgr2=0x7fb17c0d0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:23.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.303+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 shutdown_connections 2024-04-15T15:17:23.306 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:23.303+0000 7fb182dfa640 1 -- 172.21.15.70:0/4210781355 wait complete. 2024-04-15T15:17:24.316 INFO:tasks.ceph:Scrubbing osd.0 2024-04-15T15:17:24.317 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:24.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.433+0000 7f417b115640 1 Processor -- start 2024-04-15T15:17:24.553 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.433+0000 7f417b115640 1 -- start start 2024-04-15T15:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f417b115640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f41740b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f417b115640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740b38d0 con 0x7f41740ce770 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f417b115640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740b3bb0 con 0x7f41740d0470 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f417b115640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f41740b36a0 con 0x7f41740cc6b0 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f4178eb1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f41740ce770 0x7f417415c0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49778/0 (socket says 172.21.15.70:49778) 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f4178eb1640 1 -- 172.21.15.70:0/95703059 learned_addr learned my addr 172.21.15.70:0/95703059 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.434+0000 7f416bfff640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f41740b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596363326 0 0) 0x7f41740b38d0 con 0x7f41740ce770 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f415c005730 con 0x7f41740ce770 2024-04-15T15:17:24.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 453193776 0 0) 0x7f415c005730 con 0x7f41740ce770 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 msgr2=0x7f41740b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f41740b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 >> v1:172.21.15.96:6789/0 conn(0x7f41740d0470 legacy=0x7f41740ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41740b3e90 con 0x7f41740ce770 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f4170001670 con 0x7f41740ce770 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 >> v1:172.21.15.70:6789/0 conn(0x7f41740ce770 legacy=0x7f417415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f415c005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006130 0x7f415c006a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f415c006f50 0x7f415c0073d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f415c00dcb0 con 0x7f415c006f50 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f415c00de60 con 0x7f415c006130 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f415c00e060 con 0x7f41740cc6b0 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f415c006f50 0x7f415c0073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f4178eb1640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f415c005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f416bfff640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006130 0x7f415c006a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 -- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 msgr2=0x7f415c005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f41740cc6b0 0x7f415c005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 -- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006130 msgr2=0x7f415c006a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006130 0x7f415c006a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.435+0000 7f41796b2640 1 -- 172.21.15.70:0/95703059 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c005730 con 0x7f415c006f50 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.436+0000 7f41796b2640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f415c006f50 0x7f415c0073d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f416c001f60 tx=0x7f415c007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a518861b4603aa server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.436+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c001b30 con 0x7f415c006f50 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.436+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416c02d710 con 0x7f415c006f50 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.436+0000 7f416b7fe640 1 -- 172.21.15.70:0/95703059 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c02db00 con 0x7f415c006f50 2024-04-15T15:17:24.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 -- 172.21.15.70:0/95703059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f415c006f50 msgr2=0x7f415c0073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 --2- 172.21.15.70:0/95703059 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f415c006f50 0x7f415c0073d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f416c001f60 tx=0x7f415c007a40 comp rx=0 tx=0).stop 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f41796b2640 1 -- 172.21.15.70:0/95703059 reap_dead start 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 -- 172.21.15.70:0/95703059 shutdown_connections 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 -- 172.21.15.70:0/95703059 >> 172.21.15.70:0/95703059 conn(0x7f417406bbc0 msgr2=0x7f417406bfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 -- 172.21.15.70:0/95703059 shutdown_connections 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.437+0000 7f417b115640 1 -- 172.21.15.70:0/95703059 wait complete. 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.438+0000 7f417b115640 1 Processor -- start 2024-04-15T15:17:24.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.438+0000 7f417b115640 1 -- start start 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006f50 0x7f41740ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4174073790 0x7f41741f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 0x7f417415e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f417415c7d0 con 0x7f41741f3a50 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f41740ceb50 con 0x7f415c006f50 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f417b115640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f417415c950 con 0x7f4174073790 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 0x7f417415e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 0x7f417415e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51556/0 (socket says 172.21.15.70:51556) 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 -- 172.21.15.70:0/326998981 learned_addr learned my addr 172.21.15.70:0/326998981 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f4178eb1640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006f50 0x7f41740ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f416bfff640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4174073790 0x7f41741f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 -- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4174073790 msgr2=0x7f41741f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4174073790 0x7f41741f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 -- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006f50 msgr2=0x7f41740ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f415c006f50 0x7f41740ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f417415cb00 con 0x7f41741f3a50 2024-04-15T15:17:24.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.439+0000 7f41796b2640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 0x7f417415e7e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f416c002080 tx=0x7f416c0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c001b30 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f416c045070 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f417415cde0 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f416c003bb0 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4174073bf0 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.440+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f417415d300 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.441+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f416c005070 con 0x7f41741f3a50 2024-04-15T15:17:24.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f41697fa640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f416006c980 0x7f416006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f416c0ae570 con 0x7f41741f3a50 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f4178eb1640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f416006c980 0x7f416006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f41697fa640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7f4160072ec0 0x7f4160075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f416c009eb0 con 0x7f4160072ec0 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f416c0aea50 con 0x7f41741f3a50 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f416bfff640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7f4160072ec0 0x7f4160075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f4178eb1640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f416006c980 0x7f416006ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f41700010e0 tx=0x7f41700078f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.442+0000 7f416bfff640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7f4160072ec0 0x7f4160075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.443+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== osd.0 v2:172.21.15.70:6816/2850943089 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f416c009eb0 con 0x7f4160072ec0 2024-04-15T15:17:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 --> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f41740adb50 con 0x7f4160072ec0 2024-04-15T15:17:24.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f41697fa640 1 -- 172.21.15.70:0/326998981 <== osd.0 v2:172.21.15.70:6816/2850943089 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f41740adb50 con 0x7f4160072ec0 2024-04-15T15:17:24.569 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:24.569 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:24.569 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:24.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7f4160072ec0 msgr2=0x7f4160075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6816/2850943089,v1:172.21.15.70:6817/2850943089] conn(0x7f4160072ec0 0x7f4160075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f416006c980 msgr2=0x7f416006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f416006c980 0x7f416006ee40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f41700010e0 tx=0x7f41700078f0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 msgr2=0x7f417415e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f417b115640 1 --2- 172.21.15.70:0/326998981 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f41741f3a50 0x7f417415e7e0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f416c002080 tx=0x7f416c0015b0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.456+0000 7f41796b2640 1 -- 172.21.15.70:0/326998981 reap_dead start 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.457+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 shutdown_connections 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.457+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 >> 172.21.15.70:0/326998981 conn(0x7f417406bbc0 msgr2=0x7f417406f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.457+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 shutdown_connections 2024-04-15T15:17:24.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.457+0000 7f417b115640 1 -- 172.21.15.70:0/326998981 wait complete. 2024-04-15T15:17:24.573 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-15T15:17:24.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.675+0000 7fe3a4390640 1 Processor -- start 2024-04-15T15:17:24.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.675+0000 7fe3a4390640 1 -- start start 2024-04-15T15:17:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a4390640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a4390640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a4390640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe39c15db40 con 0x7fe39c0d0690 2024-04-15T15:17:24.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a4390640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fe39c15cfc0 con 0x7fe39c0af3e0 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a4390640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c15d2a0 con 0x7fe39c072d80 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a192b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a212c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a292d640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fe39c0d0690 0x7fe39c06e060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:33332/0 (socket says 172.21.15.70:33332) 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a212c640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:58702/0 (socket says 172.21.15.70:58702) 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a192b640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:33162/0 (socket says 172.21.15.70:33162) 2024-04-15T15:17:24.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.676+0000 7fe3a292d640 1 -- 172.21.15.70:0/1221260336 learned_addr learned my addr 172.21.15.70:0/1221260336 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a212c640 1 -- 172.21.15.70:0/1221260336 >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 msgr2=0x7fe39c0af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a212c640 1 --2- 172.21.15.70:0/1221260336 >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a212c640 1 -- 172.21.15.70:0/1221260336 >> v1:172.21.15.96:6789/0 conn(0x7fe39c0d0690 legacy=0x7fe39c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a212c640 1 -- 172.21.15.70:0/1221260336 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c15d580 con 0x7fe39c072d80 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a192b640 1 --2- 172.21.15.70:0/1221260336 >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a212c640 1 --2- 172.21.15.70:0/1221260336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe398009d90 tx=0x7fe39802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6aa58795898299 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.677+0000 7fe3a112a640 1 -- 172.21.15.70:0/1221260336 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398039070 con 0x7fe39c072d80 2024-04-15T15:17:24.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a112a640 1 -- 172.21.15.70:0/1221260336 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe39802cbc0 con 0x7fe39c072d80 2024-04-15T15:17:24.682 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a112a640 1 -- 172.21.15.70:0/1221260336 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398034440 con 0x7fe39c072d80 2024-04-15T15:17:24.685 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 -- 172.21.15.70:0/1221260336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 msgr2=0x7fe39c071d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 --2- 172.21.15.70:0/1221260336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe398009d90 tx=0x7fe39802be10 comp rx=0 tx=0).stop 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 -- 172.21.15.70:0/1221260336 shutdown_connections 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 --2- 172.21.15.70:0/1221260336 >> v2:172.21.15.96:3300/0 conn(0x7fe39c0af3e0 0x7fe39c0af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 --2- 172.21.15.70:0/1221260336 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c071d10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 -- 172.21.15.70:0/1221260336 >> 172.21.15.70:0/1221260336 conn(0x7fe39c06be00 msgr2=0x7fe39c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 -- 172.21.15.70:0/1221260336 shutdown_connections 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.678+0000 7fe3a4390640 1 -- 172.21.15.70:0/1221260336 wait complete. 2024-04-15T15:17:24.686 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.679+0000 7fe3a4390640 1 Processor -- start 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.679+0000 7fe3a4390640 1 -- start start 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c09c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 0x7fe39c1f61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c15cd10 con 0x7fe39c0d0690 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe39c15cb90 con 0x7fe39c0af3e0 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a4390640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe39c0d0b20 con 0x7fe39c072d80 2024-04-15T15:17:24.687 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a292d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a292d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51564/0 (socket says 172.21.15.70:51564) 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a292d640 1 -- 172.21.15.70:0/2959355479 learned_addr learned my addr 172.21.15.70:0/2959355479 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a212c640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c09c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.680+0000 7fe3a192b640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 0x7fe39c1f61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 -- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 msgr2=0x7fe39c09c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c09c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 -- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 msgr2=0x7fe39c1f61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 0x7fe39c1f61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 -- 172.21.15.70:0/2959355479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe39c0cf660 con 0x7fe39c0d0690 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a192b640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 0x7fe39c1f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a292d640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c370 tx=0x7fe39400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394017070 con 0x7fe39c0d0690 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe394005ce0 con 0x7fe39c0d0690 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3940072d0 con 0x7fe39c0d0690 2024-04-15T15:17:24.689 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe39c1f2710 con 0x7fe39c0d0690 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.681+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe39c1f2cd0 con 0x7fe39c0d0690 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.682+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe39c0689e0 con 0x7fe39c0d0690 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.682+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe39401a070 con 0x7fe39c0d0690 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.683+0000 7fe392ffd640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 0x7fe38006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.683+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fe39408ab90 con 0x7fe39c0d0690 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.683+0000 7fe3a212c640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 0x7fe38006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.690 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.683+0000 7fe3a212c640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 0x7fe38006ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe398036000 tx=0x7fe398030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.687+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe394058b30 con 0x7fe39c0d0690 2024-04-15T15:17:24.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.862+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fe39c0b2ff0 con 0x7fe38006c980 2024-04-15T15:17:24.864 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.862+0000 7fe392ffd640 1 -- 172.21.15.70:0/2959355479 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe39c0b2ff0 con 0x7fe38006c980 2024-04-15T15:17:24.865 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 0 to deep-scrub 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 msgr2=0x7fe38006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 0x7fe38006ee40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe398036000 tx=0x7fe398030000 comp rx=0 tx=0).stop 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 msgr2=0x7fe39c153e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c370 tx=0x7fe39400c830 comp rx=0 tx=0).stop 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 shutdown_connections 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe38006c980 0x7fe38006ee40 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.867 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe39c0d0690 0x7fe39c153e60 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.864+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe39c0af3e0 0x7fe39c1f61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.865+0000 7fe3a4390640 1 --2- 172.21.15.70:0/2959355479 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe39c072d80 0x7fe39c09c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.865+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 >> 172.21.15.70:0/2959355479 conn(0x7fe39c06be00 msgr2=0x7fe39c0b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.865+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 shutdown_connections 2024-04-15T15:17:24.868 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.865+0000 7fe3a4390640 1 -- 172.21.15.70:0/2959355479 wait complete. 2024-04-15T15:17:24.878 INFO:tasks.ceph:Scrubbing osd.1 2024-04-15T15:17:24.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:24.984 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.983+0000 7f24d43b5640 1 Processor -- start 2024-04-15T15:17:24.985 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.983+0000 7f24d43b5640 1 -- start start 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d43b5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d43b5640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f24cc0b1bd0 0x7f24cc0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d43b5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc0ceaf0 con 0x7f24cc0cde30 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d43b5640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f24cc0b0e50 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d43b5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc06c300 con 0x7f24cc06b9e0 2024-04-15T15:17:24.987 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d1950640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f24cc0cde30 0x7f24cc0ce250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49790/0 (socket says 172.21.15.70:49790) 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d1950640 1 -- 172.21.15.70:0/2922582431 learned_addr learned my addr 172.21.15.70:0/2922582431 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2952640 1 --2- 172.21.15.70:0/2922582431 >> v2:172.21.15.70:3300/0 conn(0x7f24cc0b1bd0 0x7f24cc0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2151640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d114f640 1 -- 172.21.15.70:0/2922582431 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 300012911 0 0) 0x7f24cc0ceaf0 con 0x7f24cc0cde30 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d114f640 1 -- 172.21.15.70:0/2922582431 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b8003520 con 0x7f24cc0cde30 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2952640 1 -- 172.21.15.70:0/2922582431 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 msgr2=0x7f24cc06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2952640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc06bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2952640 1 -- 172.21.15.70:0/2922582431 >> v1:172.21.15.70:6789/0 conn(0x7f24cc0cde30 legacy=0x7f24cc0ce250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.988 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.984+0000 7f24d2952640 1 -- 172.21.15.70:0/2922582431 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0b10f0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.989 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.985+0000 7f24d2952640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0b2060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009d90 tx=0x7f24c402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8d84253788d73c server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d114f640 1 -- 172.21.15.70:0/2922582431 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4039070 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d114f640 1 -- 172.21.15.70:0/2922582431 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c402cbc0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d114f640 1 -- 172.21.15.70:0/2922582431 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4034440 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 -- 172.21.15.70:0/2922582431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 msgr2=0x7f24cc0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0b2060 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009d90 tx=0x7f24c402be10 comp rx=0 tx=0).stop 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 -- 172.21.15.70:0/2922582431 shutdown_connections 2024-04-15T15:17:24.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0b2060 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 --2- 172.21.15.70:0/2922582431 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc06bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 -- 172.21.15.70:0/2922582431 >> 172.21.15.70:0/2922582431 conn(0x7f24cc09a9d0 msgr2=0x7f24cc0b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:24.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.987+0000 7f24d43b5640 1 -- 172.21.15.70:0/2922582431 shutdown_connections 2024-04-15T15:17:24.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.988+0000 7f24d43b5640 1 -- 172.21.15.70:0/2922582431 wait complete. 2024-04-15T15:17:24.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 Processor -- start 2024-04-15T15:17:24.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 -- start start 2024-04-15T15:17:24.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc13da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0d0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24cc0cde30 0x7f24cc13cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc1461d0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc0b16b0 con 0x7f24cc0cde30 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d43b5640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc146350 con 0x7f24cc06b9e0 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d1950640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.989+0000 7f24d1950640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51580/0 (socket says 172.21.15.70:51580) 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 -- 172.21.15.70:0/275009368 learned_addr learned my addr 172.21.15.70:0/275009368 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d2151640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc13da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.995 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d2952640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24cc0cde30 0x7f24cc13cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 -- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 msgr2=0x7f24cc13da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f24cc06b9e0 0x7f24cc13da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 -- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24cc0cde30 msgr2=0x7f24cc13cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f24cc0cde30 0x7f24cc13cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0d0da0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d1950640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0d0860 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c430 tx=0x7f24bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc017070 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24bc005ce0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.996 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc0cf170 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0072f0 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.990+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc0cf730 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.991+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f24cc0cef20 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.991+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f24bc01a070 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24c2ffd640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f24b006c980 0x7f24b006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24d2151640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f24b006c980 0x7f24b006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f24bc08ae30 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24c2ffd640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f24b0072ec0 0x7f24b0075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:24.997 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f24bc015e70 con 0x7f24b0072ec0 2024-04-15T15:17:24.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f24bc08d580 con 0x7f24cc0b1bd0 2024-04-15T15:17:24.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24d2952640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f24b0072ec0 0x7f24b0075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:24.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.992+0000 7f24d2151640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f24b006c980 0x7f24b006ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009aa0 tx=0x7f24c8009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.998 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.993+0000 7f24d2952640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f24b0072ec0 0x7f24b0075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:24.999 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:24.993+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== osd.1 v2:172.21.15.70:6800/1494795690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f24bc015e70 con 0x7f24b0072ec0 2024-04-15T15:17:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.006+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 --> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f24cc0690f0 con 0x7f24b0072ec0 2024-04-15T15:17:25.008 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.006+0000 7f24c2ffd640 1 -- 172.21.15.70:0/275009368 <== osd.1 v2:172.21.15.70:6800/1494795690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f24cc0690f0 con 0x7f24b0072ec0 2024-04-15T15:17:25.008 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:25.009 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:25.009 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f24b0072ec0 msgr2=0x7f24b0075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6800/1494795690,v1:172.21.15.70:6801/1494795690] conn(0x7f24b0072ec0 0x7f24b0075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f24b006c980 msgr2=0x7f24b006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f24b006c980 0x7f24b006ee40 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f24c8009aa0 tx=0x7f24c8009190 comp rx=0 tx=0).stop 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 msgr2=0x7f24cc0d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 --2- 172.21.15.70:0/275009368 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f24cc0b1bd0 0x7f24cc0d0860 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c430 tx=0x7f24bc00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d2952640 1 -- 172.21.15.70:0/275009368 reap_dead start 2024-04-15T15:17:25.010 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 shutdown_connections 2024-04-15T15:17:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 >> 172.21.15.70:0/275009368 conn(0x7f24cc09a9d0 msgr2=0x7f24cc06f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 shutdown_connections 2024-04-15T15:17:25.011 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.007+0000 7f24d43b5640 1 -- 172.21.15.70:0/275009368 wait complete. 2024-04-15T15:17:25.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-15T15:17:25.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.121+0000 7f30904b1640 1 Processor -- start 2024-04-15T15:17:25.123 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 -- start start 2024-04-15T15:17:25.125 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 0x7f30880d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f308815eae0 0x7f30880d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30880b47a0 con 0x7f308815e700 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f30880b53b0 con 0x7f308815eae0 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.122+0000 7f30904b1640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f30880b4a80 con 0x7f30880d2360 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308e24d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f308815e700 0x7f30880b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49800/0 (socket says 172.21.15.70:49800) 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308e24d640 1 -- 172.21.15.70:0/2465454073 learned_addr learned my addr 172.21.15.70:0/2465454073 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:25.126 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308ea4e640 1 --2- 172.21.15.70:0/2465454073 >> v2:172.21.15.70:3300/0 conn(0x7f308815eae0 0x7f30880d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308da4c640 1 --2- 172.21.15.70:0/2465454073 >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 0x7f30880d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308d24b640 1 -- 172.21.15.70:0/2465454073 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149064450 0 0) 0x7f30880b47a0 con 0x7f308815e700 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308d24b640 1 -- 172.21.15.70:0/2465454073 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3074003520 con 0x7f308815e700 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308ea4e640 1 -- 172.21.15.70:0/2465454073 >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 msgr2=0x7f30880d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308ea4e640 1 --2- 172.21.15.70:0/2465454073 >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 0x7f30880d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308ea4e640 1 -- 172.21.15.70:0/2465454073 >> v1:172.21.15.70:6789/0 conn(0x7f308815e700 legacy=0x7f30880b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308ea4e640 1 -- 172.21.15.70:0/2465454073 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30880b4d60 con 0x7f308815eae0 2024-04-15T15:17:25.127 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.123+0000 7f308da4c640 1 --2- 172.21.15.70:0/2465454073 >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 0x7f30880d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:25.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f308ea4e640 1 --2- 172.21.15.70:0/2465454073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815eae0 0x7f30880d30a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f308808c7b0 tx=0x7f308002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c293079b7b6cfce server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f308d24b640 1 -- 172.21.15.70:0/2465454073 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080039070 con 0x7f308815eae0 2024-04-15T15:17:25.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f308d24b640 1 -- 172.21.15.70:0/2465454073 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f308002cc80 con 0x7f308815eae0 2024-04-15T15:17:25.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f308d24b640 1 -- 172.21.15.70:0/2465454073 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3080034660 con 0x7f308815eae0 2024-04-15T15:17:25.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 -- 172.21.15.70:0/2465454073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815eae0 msgr2=0x7f30880d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 --2- 172.21.15.70:0/2465454073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815eae0 0x7f30880d30a0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f308808c7b0 tx=0x7f308002bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 -- 172.21.15.70:0/2465454073 shutdown_connections 2024-04-15T15:17:25.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 --2- 172.21.15.70:0/2465454073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815eae0 0x7f30880d30a0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 --2- 172.21.15.70:0/2465454073 >> v2:172.21.15.96:3300/0 conn(0x7f30880d2360 0x7f30880d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 -- 172.21.15.70:0/2465454073 >> 172.21.15.70:0/2465454073 conn(0x7f308808bf60 msgr2=0x7f30880d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.124+0000 7f30904b1640 1 -- 172.21.15.70:0/2465454073 shutdown_connections 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.125+0000 7f30904b1640 1 -- 172.21.15.70:0/2465454073 wait complete. 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 Processor -- start 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 -- start start 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 0x7f3088160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.132 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 0x7f30880ca010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f30880d1d40 con 0x7f308815e700 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f30880b4450 con 0x7f30880d2360 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f30904b1640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f30880d1ec0 con 0x7f308815eae0 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f308da4c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.126+0000 7f308da4c640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51596/0 (socket says 172.21.15.70:51596) 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 -- 172.21.15.70:0/3500760732 learned_addr learned my addr 172.21.15.70:0/3500760732 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308e24d640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 0x7f3088160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308ea4e640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 0x7f30880ca010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 -- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 msgr2=0x7f30880ca010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.133 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 0x7f30880ca010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 -- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 msgr2=0x7f3088160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 0x7f3088160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 -- 172.21.15.70:0/3500760732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f308815aea0 con 0x7f308815e700 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308e24d640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 0x7f3088160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308ea4e640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 0x7f30880ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f308da4c640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f307c00d800 tx=0x7f307c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c004030 con 0x7f308815e700 2024-04-15T15:17:25.134 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f307c0041d0 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3088160ad0 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f307c0096a0 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.127+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3088161090 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.128+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30880689e0 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.129+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f307c009840 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.129+0000 7f307affd640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 0x7f306c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.129+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f307c08bc90 con 0x7f308815e700 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.129+0000 7f308e24d640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 0x7f306c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.135 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.130+0000 7f308e24d640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 0x7f306c06ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3084001360 tx=0x7f3084009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.136 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.133+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f307c05a280 con 0x7f308815e700 2024-04-15T15:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.309+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f30880ce9b0 con 0x7f306c06c980 2024-04-15T15:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.309+0000 7f307affd640 1 -- 172.21.15.70:0/3500760732 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f30880ce9b0 con 0x7f306c06c980 2024-04-15T15:17:25.311 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 1 to deep-scrub 2024-04-15T15:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 msgr2=0x7f306c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 0x7f306c06ee40 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3084001360 tx=0x7f3084009040 comp rx=0 tx=0).stop 2024-04-15T15:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 msgr2=0x7f30880c9ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.313 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f307c00d800 tx=0x7f307c00dcc0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 shutdown_connections 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f308815eae0 0x7f30880ca010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f306c06c980 0x7f306c06ee40 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f308815e700 0x7f30880c9ad0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 --2- 172.21.15.70:0/3500760732 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f30880d2360 0x7f3088160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 >> 172.21.15.70:0/3500760732 conn(0x7f308808bf60 msgr2=0x7f30880b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.314 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 shutdown_connections 2024-04-15T15:17:25.315 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.311+0000 7f30904b1640 1 -- 172.21.15.70:0/3500760732 wait complete. 2024-04-15T15:17:25.324 INFO:tasks.ceph:Scrubbing osd.2 2024-04-15T15:17:25.324 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:25.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.429+0000 7f570b59e640 1 Processor -- start 2024-04-15T15:17:25.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.429+0000 7f570b59e640 1 -- start start 2024-04-15T15:17:25.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570b59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f570c0ce500 0x7f570c0d2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570b59e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c0d0560 con 0x7f570c0ce120 2024-04-15T15:17:25.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570b59e640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c06d750 con 0x7f570c09b0e0 2024-04-15T15:17:25.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570b59e640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f570c0cc8c0 con 0x7f570c0ce500 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f5709d9b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f570c0ce500 0x7f570c0d2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f570c0ce120 0x7f570c09a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49804/0 (socket says 172.21.15.70:49804) 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.430+0000 7f570a59c640 1 -- 172.21.15.70:0/1770489576 learned_addr learned my addr 172.21.15.70:0/1770489576 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f5709d9b640 1 -- 172.21.15.70:0/1770489576 >> v1:172.21.15.96:6789/0 conn(0x7f570c09b0e0 legacy=0x7f570c0d0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f5709d9b640 1 -- 172.21.15.70:0/1770489576 >> v1:172.21.15.70:6789/0 conn(0x7f570c0ce120 legacy=0x7f570c09a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f5709d9b640 1 -- 172.21.15.70:0/1770489576 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c06da30 con 0x7f570c0ce500 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f570959a640 1 -- 172.21.15.70:0/1770489576 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4178878921 0 0) 0x7f570c0d0560 con 0x7f570c0ce120 2024-04-15T15:17:25.434 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f5709d9b640 1 --2- 172.21.15.70:0/1770489576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce500 0x7f570c0d2d90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f57000097a0 tx=0x7f570002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eefb20d396677ee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.431+0000 7f570959a640 1 -- 172.21.15.70:0/1770489576 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700035020 con 0x7f570c0ce500 2024-04-15T15:17:25.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570959a640 1 -- 172.21.15.70:0/1770489576 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5700031830 con 0x7f570c0ce500 2024-04-15T15:17:25.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570959a640 1 -- 172.21.15.70:0/1770489576 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700031c00 con 0x7f570c0ce500 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 -- 172.21.15.70:0/1770489576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce500 msgr2=0x7f570c0d2d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 --2- 172.21.15.70:0/1770489576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce500 0x7f570c0d2d90 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f57000097a0 tx=0x7f570002b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 -- 172.21.15.70:0/1770489576 shutdown_connections 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 --2- 172.21.15.70:0/1770489576 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce500 0x7f570c0d2d90 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 -- 172.21.15.70:0/1770489576 >> 172.21.15.70:0/1770489576 conn(0x7f570c0d2270 msgr2=0x7f570c0d2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 -- 172.21.15.70:0/1770489576 shutdown_connections 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.432+0000 7f570b59e640 1 -- 172.21.15.70:0/1770489576 wait complete. 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.433+0000 7f570b59e640 1 Processor -- start 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.433+0000 7f570b59e640 1 -- start start 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f570c09b0e0 0x7f570c1f4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 0x7f570c0c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f570c0ce500 0x7f570c15d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f570c06dd10 con 0x7f570c0ce120 2024-04-15T15:17:25.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f570c06d500 con 0x7f570c0ce500 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f570b59e640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f570c06de90 con 0x7f570c09b0e0 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f5709d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 0x7f570c0c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f5709d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 0x7f570c0c7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51612/0 (socket says 172.21.15.70:51612) 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.434+0000 7f5709d9b640 1 -- 172.21.15.70:0/2539072857 learned_addr learned my addr 172.21.15.70:0/2539072857 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570a59c640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f570c09b0e0 0x7f570c1f4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570ad9d640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f570c0ce500 0x7f570c15d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 -- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f570c09b0e0 msgr2=0x7f570c1f4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f570c09b0e0 0x7f570c1f4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 -- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f570c0ce500 msgr2=0x7f570c15d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f570c0ce500 0x7f570c15d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c0c7040 con 0x7f570c0ce120 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570ad9d640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f570c0ce500 0x7f570c15d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570a59c640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f570c09b0e0 0x7f570c1f4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:25.442 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f5709d9b640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 0x7f570c0c7e00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f57000097a0 tx=0x7f57000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700004230 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57000043f0 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c0c7320 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700003850 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.435+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c0c78e0 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.436+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f570c0c6df0 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.436+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f570004c060 con 0x7f570c0ce120 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f56faffd640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f56ec06c980 0x7f56ec06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.443 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f57000ad720 con 0x7f570c0ce120 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f570a59c640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f56ec06c980 0x7f56ec06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f56faffd640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f56ec072ec0 0x7f56ec075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f570004fe50 con 0x7f56ec072ec0 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f57000b4050 con 0x7f570c0ce120 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f570ad9d640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f56ec072ec0 0x7f56ec075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f570a59c640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f56ec06c980 0x7f56ec06ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0012e0 tx=0x7f56fc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.444 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.437+0000 7f570ad9d640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f56ec072ec0 0x7f56ec075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.445 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.438+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== osd.2 v2:172.21.15.70:6808/3050098076 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f570004fe50 con 0x7f56ec072ec0 2024-04-15T15:17:25.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.451+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 --> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f570c04fdc0 con 0x7f56ec072ec0 2024-04-15T15:17:25.453 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.451+0000 7f56faffd640 1 -- 172.21.15.70:0/2539072857 <== osd.2 v2:172.21.15.70:6808/3050098076 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f570c04fdc0 con 0x7f56ec072ec0 2024-04-15T15:17:25.453 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:25.453 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:25.454 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:25.454 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.451+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f56ec072ec0 msgr2=0x7f56ec075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6808/3050098076,v1:172.21.15.70:6809/3050098076] conn(0x7f56ec072ec0 0x7f56ec075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f56ec06c980 msgr2=0x7f56ec06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f56ec06c980 0x7f56ec06ee40 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0012e0 tx=0x7f56fc009040 comp rx=0 tx=0).stop 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 msgr2=0x7f570c0c7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 --2- 172.21.15.70:0/2539072857 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f570c0ce120 0x7f570c0c7e00 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f57000097a0 tx=0x7f57000046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570ad9d640 1 -- 172.21.15.70:0/2539072857 reap_dead start 2024-04-15T15:17:25.455 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 shutdown_connections 2024-04-15T15:17:25.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 >> 172.21.15.70:0/2539072857 conn(0x7f570c0d2270 msgr2=0x7f570c1616d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 shutdown_connections 2024-04-15T15:17:25.456 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.452+0000 7f570b59e640 1 -- 172.21.15.70:0/2539072857 wait complete. 2024-04-15T15:17:25.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-15T15:17:25.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.567+0000 7f04cd796640 1 Processor -- start 2024-04-15T15:17:25.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.567+0000 7f04cd796640 1 -- start start 2024-04-15T15:17:25.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.567+0000 7f04cd796640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04c815e790 0x7f04c80d3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04cd796640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c80cfd20 0x7f04c80d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04cd796640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f04c80d0f00 0x7f04c80d13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04cd796640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f04c80d2db0 con 0x7f04c815e790 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04cd796640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f04c80d2480 con 0x7f04c80d0f00 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04cd796640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f04c80d2760 con 0x7f04c80cfd20 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04c815e790 0x7f04c80d3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f04c815e790 0x7f04c80d3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51614/0 (socket says 172.21.15.70:51614) 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 -- 172.21.15.70:0/4007749247 learned_addr learned my addr 172.21.15.70:0/4007749247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 -- 172.21.15.70:0/4007749247 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c80cfd20 msgr2=0x7f04c80d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/4007749247 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c80cfd20 0x7f04c80d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 -- 172.21.15.70:0/4007749247 >> v2:172.21.15.96:3300/0 conn(0x7f04c80d0f00 msgr2=0x7f04c80d13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c77fe640 1 --2- 172.21.15.70:0/4007749247 >> v2:172.21.15.96:3300/0 conn(0x7f04c80d0f00 0x7f04c80d13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/4007749247 >> v2:172.21.15.96:3300/0 conn(0x7f04c80d0f00 0x7f04c80d13b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.568+0000 7f04c6ffd640 1 -- 172.21.15.70:0/4007749247 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c80d2a40 con 0x7f04c815e790 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/4007749247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c815e790 0x7f04c80d3240 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f04c80cec80 tx=0x7f04b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640e31f3afabb9c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04c67fc640 1 -- 172.21.15.70:0/4007749247 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8035020 con 0x7f04c815e790 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04c67fc640 1 -- 172.21.15.70:0/4007749247 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b8031870 con 0x7f04c815e790 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04c67fc640 1 -- 172.21.15.70:0/4007749247 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8031c40 con 0x7f04c815e790 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 -- 172.21.15.70:0/4007749247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c815e790 msgr2=0x7f04c80d3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.573 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 --2- 172.21.15.70:0/4007749247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c815e790 0x7f04c80d3240 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f04c80cec80 tx=0x7f04b802b910 comp rx=0 tx=0).stop 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 -- 172.21.15.70:0/4007749247 shutdown_connections 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 --2- 172.21.15.70:0/4007749247 >> v2:172.21.15.96:3300/0 conn(0x7f04c80d0f00 0x7f04c80d13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 --2- 172.21.15.70:0/4007749247 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c80cfd20 0x7f04c80d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 --2- 172.21.15.70:0/4007749247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c815e790 0x7f04c80d3240 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 -- 172.21.15.70:0/4007749247 >> 172.21.15.70:0/4007749247 conn(0x7f04c808bf60 msgr2=0x7f04c80d46a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 -- 172.21.15.70:0/4007749247 shutdown_connections 2024-04-15T15:17:25.574 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.569+0000 7f04cd796640 1 -- 172.21.15.70:0/4007749247 wait complete. 2024-04-15T15:17:25.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 Processor -- start 2024-04-15T15:17:25.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 -- start start 2024-04-15T15:17:25.577 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 0x7f04c809e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 0x7f04c80b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f04c815ecf0 con 0x7f04c80cfd20 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f04c815eb70 con 0x7f04c80d0f00 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.571+0000 7f04cd796640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f04c80b74f0 con 0x7f04c815e790 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51630/0 (socket says 172.21.15.70:51630) 2024-04-15T15:17:25.578 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 -- 172.21.15.70:0/1551035043 learned_addr learned my addr 172.21.15.70:0/1551035043 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c77fe640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 0x7f04c80b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04be7fc640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 0x7f04c809e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 -- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 msgr2=0x7f04c80b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 0x7f04c80b4b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 -- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 msgr2=0x7f04c809e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 0x7f04c809e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 -- 172.21.15.70:0/1551035043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c80a0090 con 0x7f04c80cfd20 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04be7fc640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 0x7f04c809e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:25.579 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c77fe640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 0x7f04c80b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04c6ffd640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f04b802b950 tx=0x7f04b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8040070 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04b803a420 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c80a0370 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8035040 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.572+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c80b22c0 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.573+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04c80a0760 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.574+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f04b804b060 con 0x7f04c80cfd20 2024-04-15T15:17:25.580 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.574+0000 7f04bffff640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 0x7f04a806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.574+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f04b80ad9a0 con 0x7f04c80cfd20 2024-04-15T15:17:25.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.574+0000 7f04be7fc640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 0x7f04a806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.574+0000 7f04be7fc640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 0x7f04a806ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f04b0006fd0 tx=0x7f04b0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.581 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.578+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f04b807bf90 con 0x7f04c80cfd20 2024-04-15T15:17:25.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.754+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f04c80a0540 con 0x7f04a806c980 2024-04-15T15:17:25.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.755+0000 7f04bffff640 1 -- 172.21.15.70:0/1551035043 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f04c80a0540 con 0x7f04a806c980 2024-04-15T15:17:25.757 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 2 to deep-scrub 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 msgr2=0x7f04a806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 0x7f04a806ee40 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f04b0006fd0 tx=0x7f04b0008040 comp rx=0 tx=0).stop 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 msgr2=0x7f04c80b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f04b802b950 tx=0x7f04b80046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 shutdown_connections 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f04c815e790 0x7f04c80b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f04c80d0f00 0x7f04c809e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f04a806c980 0x7f04a806ee40 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 --2- 172.21.15.70:0/1551035043 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f04c80cfd20 0x7f04c80b1cd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 >> 172.21.15.70:0/1551035043 conn(0x7f04c808bf60 msgr2=0x7f04c80cd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 shutdown_connections 2024-04-15T15:17:25.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.757+0000 7f04cd796640 1 -- 172.21.15.70:0/1551035043 wait complete. 2024-04-15T15:17:25.769 INFO:tasks.ceph:Scrubbing osd.3 2024-04-15T15:17:25.770 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.875+0000 7fc4761f0640 1 Processor -- start 2024-04-15T15:17:25.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.875+0000 7fc4761f0640 1 -- start start 2024-04-15T15:17:25.880 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc4761f0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4700d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc4761f0640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fc4700d13c0 0x7fc47006c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc4761f0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4700d03c0 con 0x7fc47015e8c0 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc4761f0640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fc47015eca0 con 0x7fc4700d13c0 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc4761f0640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc4700d1830 con 0x7fc4700d3100 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fc47015e8c0 0x7fc4700cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49806/0 (socket says 172.21.15.70:49806) 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46f7fe640 1 -- 172.21.15.70:0/3857269319 learned_addr learned my addr 172.21.15.70:0/3857269319 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> v2:172.21.15.96:3300/0 conn(0x7fc4700d13c0 0x7fc47006c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.881 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46effd640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4700d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3934541949 0 0) 0x7fc4700d03c0 con 0x7fc47015e8c0 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc458003520 con 0x7fc47015e8c0 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 8424962 0 0) 0x7fc458003520 con 0x7fc47015e8c0 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 msgr2=0x7fc4700d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4700d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 >> v2:172.21.15.96:3300/0 conn(0x7fc4700d13c0 msgr2=0x7fc47006c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 --2- 172.21.15.70:0/3857269319 >> v2:172.21.15.96:3300/0 conn(0x7fc4700d13c0 0x7fc47006c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.876+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4700d2340 con 0x7fc47015e8c0 2024-04-15T15:17:25.882 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7fc460001670 con 0x7fc47015e8c0 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> v2:172.21.15.96:3300/0 conn(0x7fc4700d13c0 0x7fc47006c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 >> v1:172.21.15.70:6789/0 conn(0x7fc47015e8c0 legacy=0x7fc4700cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc458003f30 0x7fc458004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc458004d40 0x7fc4580051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc47015eca0 con 0x7fc458004d40 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4700d1830 con 0x7fc458003f30 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc45800b9c0 con 0x7fc4700d3100 2024-04-15T15:17:25.883 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc458004d40 0x7fc4580051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46f7fe640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46effd640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc458003f30 0x7fc458004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 -- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 msgr2=0x7fc4580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700d3100 0x7fc4580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 -- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc458003f30 msgr2=0x7fc458004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc458003f30 0x7fc458004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.877+0000 7fc46ffff640 1 -- 172.21.15.70:0/3857269319 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc458003520 con 0x7fc458004d40 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc458004d40 0x7fc4580051c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e350 tx=0x7fc458005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d7a1b7461769be server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.884 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468039070 con 0x7fc458004d40 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc46802cdf0 con 0x7fc458004d40 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc46e7fc640 1 -- 172.21.15.70:0/3857269319 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc468009de0 con 0x7fc458004d40 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc4761f0640 1 -- 172.21.15.70:0/3857269319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc458004d40 msgr2=0x7fc4580051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc4761f0640 1 --2- 172.21.15.70:0/3857269319 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc458004d40 0x7fc4580051c0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fc46800e350 tx=0x7fc458005830 comp rx=0 tx=0).stop 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc46ffff640 1 -- 172.21.15.70:0/3857269319 reap_dead start 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc4761f0640 1 -- 172.21.15.70:0/3857269319 shutdown_connections 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.878+0000 7fc4761f0640 1 -- 172.21.15.70:0/3857269319 >> 172.21.15.70:0/3857269319 conn(0x7fc47008bf60 msgr2=0x7fc4700d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.879+0000 7fc4761f0640 1 -- 172.21.15.70:0/3857269319 shutdown_connections 2024-04-15T15:17:25.885 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.879+0000 7fc4761f0640 1 -- 172.21.15.70:0/3857269319 wait complete. 2024-04-15T15:17:25.888 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.880+0000 7fc4761f0640 1 Processor -- start 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.880+0000 7fc4761f0640 1 -- start start 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 0x7fc4701f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700c8650 0x7fc4700c8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4701edbd0 0x7fc4701ee050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc4700d2900 con 0x7fc4701f7a40 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fc4700d2be0 con 0x7fc4701edbd0 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fc4700d4de0 con 0x7fc4700c8650 2024-04-15T15:17:25.889 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 0x7fc4701f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 0x7fc4701f7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51648/0 (socket says 172.21.15.70:51648) 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 -- 172.21.15.70:0/3191861324 learned_addr learned my addr 172.21.15.70:0/3191861324 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 -- 172.21.15.70:0/3191861324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700c8650 msgr2=0x7fc4700c8ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fc4700c8650 0x7fc4700c8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 -- 172.21.15.70:0/3191861324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4701edbd0 msgr2=0x7fc4701ee050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fc4701edbd0 0x7fc4701ee050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4701f8360 con 0x7fc4701f7a40 2024-04-15T15:17:25.890 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc46f7fe640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 0x7fc4701f7e20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc4600072f0 tx=0x7fc4600077b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46001a070 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc46000fa00 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4700c9130 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46000fdf0 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.881+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4701567e0 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.882+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc4700689e0 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fc46001d070 con 0x7fc4701f7a40 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc45c06c980 0x7fc45c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.891 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fc46008b650 con 0x7fc4701f7a40 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc46effd640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc45c06c980 0x7fc45c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7fc45c072ec0 0x7fc45c075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc45c0759d0 con 0x7fc45c072ec0 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fc460092050 con 0x7fc4701f7a40 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.883+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7fc45c072ec0 0x7fc45c075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.884+0000 7fc46effd640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc45c06c980 0x7fc45c06ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4700ba4c0 tx=0x7fc464009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.884+0000 7fc46ffff640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7fc45c072ec0 0x7fc45c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:25.892 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.884+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== osd.3 v2:172.21.15.70:6824/1402304147 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc45c0759d0 con 0x7fc45c072ec0 2024-04-15T15:17:25.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.897+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 --> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc4700cfc90 con 0x7fc45c072ec0 2024-04-15T15:17:25.899 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.897+0000 7fc457fff640 1 -- 172.21.15.70:0/3191861324 <== osd.3 v2:172.21.15.70:6824/1402304147 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc4700cfc90 con 0x7fc45c072ec0 2024-04-15T15:17:25.899 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:25.900 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:25.900 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:25.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7fc45c072ec0 msgr2=0x7fc45c075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6824/1402304147,v1:172.21.15.70:6825/1402304147] conn(0x7fc45c072ec0 0x7fc45c075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc45c06c980 msgr2=0x7fc45c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.901 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fc45c06c980 0x7fc45c06ee40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc4700ba4c0 tx=0x7fc464009320 comp rx=0 tx=0).stop 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 msgr2=0x7fc4701f7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 --2- 172.21.15.70:0/3191861324 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc4701f7a40 0x7fc4701f7e20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fc4600072f0 tx=0x7fc4600077b0 comp rx=0 tx=0).stop 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc46ffff640 1 -- 172.21.15.70:0/3191861324 reap_dead start 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 shutdown_connections 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 >> 172.21.15.70:0/3191861324 conn(0x7fc47008bf60 msgr2=0x7fc47015d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 shutdown_connections 2024-04-15T15:17:25.902 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:25.898+0000 7fc4761f0640 1 -- 172.21.15.70:0/3191861324 wait complete. 2024-04-15T15:17:25.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-15T15:17:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.013+0000 7ff9c107b640 1 Processor -- start 2024-04-15T15:17:26.015 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.013+0000 7ff9c107b640 1 -- start start 2024-04-15T15:17:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9c107b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc0d2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9c107b640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9c107b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9bc0b47a0 con 0x7ff9bc15e700 2024-04-15T15:17:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9c107b640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7ff9bc0b53b0 con 0x7ff9bc15eae0 2024-04-15T15:17:26.017 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9c107b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc0b4a80 con 0x7ff9bc0d2360 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bb59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7ff9bc15e700 0x7ff9bc0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49820/0 (socket says 172.21.15.70:49820) 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bb59e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51664/0 (socket says 172.21.15.70:51664) 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bad9d640 1 -- 172.21.15.70:0/3892549803 learned_addr learned my addr 172.21.15.70:0/3892549803 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9ba59c640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc0d2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bb59e640 1 -- 172.21.15.70:0/3892549803 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 msgr2=0x7ff9bc0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc0d2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.018 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.014+0000 7ff9bb59e640 1 -- 172.21.15.70:0/3892549803 >> v1:172.21.15.70:6789/0 conn(0x7ff9bc15e700 legacy=0x7ff9bc0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9bb59e640 1 -- 172.21.15.70:0/3892549803 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc0b4d60 con 0x7ff9bc15eae0 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9ba59c640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097a0 tx=0x7ff9b402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ed5a9fd7ff8c89 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9b9d9b640 1 -- 172.21.15.70:0/3892549803 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4035020 con 0x7ff9bc15eae0 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9b9d9b640 1 -- 172.21.15.70:0/3892549803 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b4031830 con 0x7ff9bc15eae0 2024-04-15T15:17:26.019 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.015+0000 7ff9b9d9b640 1 -- 172.21.15.70:0/3892549803 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4031c00 con 0x7ff9bc15eae0 2024-04-15T15:17:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 -- 172.21.15.70:0/3892549803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 msgr2=0x7ff9bc0d30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff9b40097a0 tx=0x7ff9b402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 -- 172.21.15.70:0/3892549803 shutdown_connections 2024-04-15T15:17:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0d30a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.022 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 --2- 172.21.15.70:0/3892549803 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc0d2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 -- 172.21.15.70:0/3892549803 >> 172.21.15.70:0/3892549803 conn(0x7ff9bc08bf60 msgr2=0x7ff9bc0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 -- 172.21.15.70:0/3892549803 shutdown_connections 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.016+0000 7ff9c107b640 1 -- 172.21.15.70:0/3892549803 wait complete. 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.017+0000 7ff9c107b640 1 Processor -- start 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.017+0000 7ff9c107b640 1 -- start start 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc1618c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9bc15e700 0x7ff9bc0c9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.023 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0d1d40 con 0x7ff9bc15eae0 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7ff9bc0b4450 con 0x7ff9bc15e700 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9c107b640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7ff9bc0d1ec0 con 0x7ff9bc0d2360 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51680/0 (socket says 172.21.15.70:51680) 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 -- 172.21.15.70:0/1646670849 learned_addr learned my addr 172.21.15.70:0/1646670849 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 -- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 msgr2=0x7ff9bc1618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc1618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.024 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9bb59e640 1 -- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9bc15e700 msgr2=0x7ff9bc0c9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.018+0000 7ff9ba59c640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9bc15e700 0x7ff9bc0c9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9bc15e700 0x7ff9bc0c9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9bb59e640 1 -- 172.21.15.70:0/1646670849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9bc161e00 con 0x7ff9bc15eae0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9bb59e640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4034040 tx=0x7ff9b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4040070 con 0x7ff9bc15eae0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9b40398c0 con 0x7ff9bc15eae0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9bc0c99a0 con 0x7ff9bc15eae0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9b4035040 con 0x7ff9bc15eae0 2024-04-15T15:17:26.025 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.019+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9bc1f7b10 con 0x7ff9bc15eae0 2024-04-15T15:17:26.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.020+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9bc0689e0 con 0x7ff9bc15eae0 2024-04-15T15:17:26.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.020+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7ff9b4038020 con 0x7ff9bc15eae0 2024-04-15T15:17:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.021+0000 7ff9b37fe640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 0x7ff99c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.021+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7ff9b40ad400 con 0x7ff9bc15eae0 2024-04-15T15:17:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.021+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 0x7ff99c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.021+0000 7ff9bad9d640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 0x7ff99c06ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac001380 tx=0x7ff9ac007c70 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.025+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7ff9b407b960 con 0x7ff9bc15eae0 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.203+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7ff9bc0ba500 con 0x7ff99c06c980 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.203+0000 7ff9b37fe640 1 -- 172.21.15.70:0/1646670849 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff9bc0ba500 con 0x7ff99c06c980 2024-04-15T15:17:26.205 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 3 to deep-scrub 2024-04-15T15:17:26.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 msgr2=0x7ff99c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 0x7ff99c06ee40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac001380 tx=0x7ff9ac007c70 comp rx=0 tx=0).stop 2024-04-15T15:17:26.207 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 msgr2=0x7ff9bc0b2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7ff9b4034040 tx=0x7ff9b40046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 shutdown_connections 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7ff99c06c980 0x7ff99c06ee40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff9bc15eae0 0x7ff9bc0b2550 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7ff9bc15e700 0x7ff9bc0c9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 --2- 172.21.15.70:0/1646670849 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7ff9bc0d2360 0x7ff9bc1618c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 >> 172.21.15.70:0/1646670849 conn(0x7ff9bc08bf60 msgr2=0x7ff9bc15a0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.208 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 shutdown_connections 2024-04-15T15:17:26.209 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.205+0000 7ff9c107b640 1 -- 172.21.15.70:0/1646670849 wait complete. 2024-04-15T15:17:26.218 INFO:tasks.ceph:Scrubbing osd.4 2024-04-15T15:17:26.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:26.324 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.323+0000 7f95746f2640 1 Processor -- start 2024-04-15T15:17:26.325 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.323+0000 7f95746f2640 1 -- start start 2024-04-15T15:17:26.327 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f95746f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f956c0add10 0x7f956c162dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f95746f2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f956c0b59a0 0x7f956c0b5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f95746f2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956c15f500 con 0x7f956c0ae0f0 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f95746f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f956c15f7e0 con 0x7f956c0add10 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f95746f2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f956c15fac0 con 0x7f956c0b59a0 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f956c0add10 0x7f956c162dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f956c0add10 0x7f956c162dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51696/0 (socket says 172.21.15.70:51696) 2024-04-15T15:17:26.328 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 -- 172.21.15.70:0/449497206 learned_addr learned my addr 172.21.15.70:0/449497206 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f9572c8f640 1 --2- 172.21.15.70:0/449497206 >> v2:172.21.15.96:3300/0 conn(0x7f956c0b59a0 0x7f956c0b5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 -- 172.21.15.70:0/449497206 >> v2:172.21.15.96:3300/0 conn(0x7f956c0b59a0 msgr2=0x7f956c0b5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 --2- 172.21.15.70:0/449497206 >> v2:172.21.15.96:3300/0 conn(0x7f956c0b59a0 0x7f956c0b5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 -- 172.21.15.70:0/449497206 >> v1:172.21.15.96:6789/0 conn(0x7f956c0ae0f0 legacy=0x7f956c15eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.324+0000 7f957248e640 1 -- 172.21.15.70:0/449497206 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c0b5480 con 0x7f956c0add10 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.325+0000 7f957248e640 1 --2- 172.21.15.70:0/449497206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c162dd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f956c0b4f40 tx=0x7f956802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f663c495c673643b server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.325+0000 7f957148c640 1 -- 172.21.15.70:0/449497206 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956802ca80 con 0x7f956c0add10 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.325+0000 7f957148c640 1 -- 172.21.15.70:0/449497206 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f956802cc20 con 0x7f956c0add10 2024-04-15T15:17:26.329 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.325+0000 7f957148c640 1 -- 172.21.15.70:0/449497206 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95680343f0 con 0x7f956c0add10 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 -- 172.21.15.70:0/449497206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 msgr2=0x7f956c162dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 --2- 172.21.15.70:0/449497206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c162dd0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f956c0b4f40 tx=0x7f956802bee0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 -- 172.21.15.70:0/449497206 shutdown_connections 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 --2- 172.21.15.70:0/449497206 >> v2:172.21.15.96:3300/0 conn(0x7f956c0b59a0 0x7f956c0b5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 --2- 172.21.15.70:0/449497206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c162dd0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 -- 172.21.15.70:0/449497206 >> 172.21.15.70:0/449497206 conn(0x7f956c06b9e0 msgr2=0x7f956c06bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 -- 172.21.15.70:0/449497206 shutdown_connections 2024-04-15T15:17:26.330 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.326+0000 7f95746f2640 1 -- 172.21.15.70:0/449497206 wait complete. 2024-04-15T15:17:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.327+0000 7f95746f2640 1 Processor -- start 2024-04-15T15:17:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.327+0000 7f95746f2640 1 -- start start 2024-04-15T15:17:26.334 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ae0f0 0x7f956c146100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f956c0b59a0 0x7f956c15bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f956c0ae550 con 0x7f956c0add10 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f956c0b5170 con 0x7f956c0ae0f0 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f956c0b6330 con 0x7f956c0b59a0 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51706/0 (socket says 172.21.15.70:51706) 2024-04-15T15:17:26.335 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 -- 172.21.15.70:0/831872069 learned_addr learned my addr 172.21.15.70:0/831872069 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9572c8f640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f956c0b59a0 0x7f956c15bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9571c8d640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ae0f0 0x7f956c146100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 -- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f956c0b59a0 msgr2=0x7f956c15bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f956c0b59a0 0x7f956c15bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 -- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ae0f0 msgr2=0x7f956c146100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ae0f0 0x7f956c146100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c160b10 con 0x7f956c0add10 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9572c8f640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f956c0b59a0 0x7f956c15bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:26.336 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9571c8d640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f956c0ae0f0 0x7f956c146100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f957248e640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c142560 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9568000f80 tx=0x7f9568004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568043020 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f956803e450 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9568009510 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c15b110 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.328+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c146640 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.329+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f956c0689e0 con 0x7f956c0add10 2024-04-15T15:17:26.337 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9568046020 con 0x7f956c0add10 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f955006c980 0x7f955006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f95680ae310 con 0x7f956c0add10 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9571c8d640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f955006c980 0x7f955006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9550072ec0 0x7f9550075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f956803ae50 con 0x7f9550072ec0 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.330+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f95680b5050 con 0x7f956c0add10 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.331+0000 7f9572c8f640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9550072ec0 0x7f9550075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.338 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.331+0000 7f9571c8d640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f955006c980 0x7f955006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f956c159630 tx=0x7f955c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.331+0000 7f9572c8f640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9550072ec0 0x7f9550075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.339 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.332+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== osd.4 v2:172.21.15.96:6800/290497787 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f956803ae50 con 0x7f9550072ec0 2024-04-15T15:17:26.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.345+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 --> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f956c0adaf0 con 0x7f9550072ec0 2024-04-15T15:17:26.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.345+0000 7f9562ffd640 1 -- 172.21.15.70:0/831872069 <== osd.4 v2:172.21.15.96:6800/290497787 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f956c0adaf0 con 0x7f9550072ec0 2024-04-15T15:17:26.348 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:26.348 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:26.348 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:26.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9550072ec0 msgr2=0x7f9550075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.96:6800/290497787,v1:172.21.15.96:6801/290497787] conn(0x7f9550072ec0 0x7f9550075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f955006c980 msgr2=0x7f955006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f955006c980 0x7f955006ee40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f956c159630 tx=0x7f955c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 msgr2=0x7f956c142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 --2- 172.21.15.70:0/831872069 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f956c0add10 0x7f956c142560 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f9568000f80 tx=0x7f9568004a40 comp rx=0 tx=0).stop 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f9572c8f640 1 -- 172.21.15.70:0/831872069 reap_dead start 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 shutdown_connections 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 >> 172.21.15.70:0/831872069 conn(0x7f956c06b9e0 msgr2=0x7f956c06f780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 shutdown_connections 2024-04-15T15:17:26.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.346+0000 7f95746f2640 1 -- 172.21.15.70:0/831872069 wait complete. 2024-04-15T15:17:26.358 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-15T15:17:26.462 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 Processor -- start 2024-04-15T15:17:26.463 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 -- start start 2024-04-15T15:17:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92140b3e00 0x7f92140729b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.464 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f92140d0570 0x7f92140d0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f921406fab0 con 0x7f921406f6d0 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f92140b49a0 con 0x7f92140b3e00 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.461+0000 7f9219dbd640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f92140b43e0 con 0x7f92140d0570 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9212ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f92140b3e00 0x7f92140729b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f92137fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f921406f6d0 0x7f921415f5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49832/0 (socket says 172.21.15.70:49832) 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f92137fe640 1 -- 172.21.15.70:0/1302415092 learned_addr learned my addr 172.21.15.70:0/1302415092 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9213fff640 1 --2- 172.21.15.70:0/1302415092 >> v2:172.21.15.96:3300/0 conn(0x7f92140d0570 0x7f92140d0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.465 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9212ffd640 1 -- 172.21.15.70:0/1302415092 >> v2:172.21.15.96:3300/0 conn(0x7f92140d0570 msgr2=0x7f92140d0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9212ffd640 1 --2- 172.21.15.70:0/1302415092 >> v2:172.21.15.96:3300/0 conn(0x7f92140d0570 0x7f92140d0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9212ffd640 1 -- 172.21.15.70:0/1302415092 >> v1:172.21.15.70:6789/0 conn(0x7f921406f6d0 legacy=0x7f921415f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f92127fc640 1 -- 172.21.15.70:0/1302415092 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 689100610 0 0) 0x7f9204001110 con 0x7f921406f6d0 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.462+0000 7f9212ffd640 1 -- 172.21.15.70:0/1302415092 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92140b46c0 con 0x7f92140b3e00 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9212ffd640 1 --2- 172.21.15.70:0/1302415092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92140b3e00 0x7f92140729b0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f92140b5a00 tx=0x7f91fc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5297c34488cbda9e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f92127fc640 1 -- 172.21.15.70:0/1302415092 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc0315a0 con 0x7f92140b3e00 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f92127fc640 1 -- 172.21.15.70:0/1302415092 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91fc037070 con 0x7f92140b3e00 2024-04-15T15:17:26.466 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f92127fc640 1 -- 172.21.15.70:0/1302415092 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91fc031960 con 0x7f92140b3e00 2024-04-15T15:17:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 -- 172.21.15.70:0/1302415092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92140b3e00 msgr2=0x7f92140729b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 --2- 172.21.15.70:0/1302415092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92140b3e00 0x7f92140729b0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f92140b5a00 tx=0x7f91fc02b840 comp rx=0 tx=0).stop 2024-04-15T15:17:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 -- 172.21.15.70:0/1302415092 shutdown_connections 2024-04-15T15:17:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 --2- 172.21.15.70:0/1302415092 >> v2:172.21.15.96:3300/0 conn(0x7f92140d0570 0x7f92140d0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.469 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 --2- 172.21.15.70:0/1302415092 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f92140b3e00 0x7f92140729b0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 -- 172.21.15.70:0/1302415092 >> 172.21.15.70:0/1302415092 conn(0x7f921406b9e0 msgr2=0x7f921406bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 -- 172.21.15.70:0/1302415092 shutdown_connections 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.463+0000 7f9219dbd640 1 -- 172.21.15.70:0/1302415092 wait complete. 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 Processor -- start 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 -- start start 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92140b3e00 0x7f92141e7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f92140d0570 0x7f921414a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.470 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9214072ef0 con 0x7f921406f6d0 2024-04-15T15:17:26.471 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9214072490 con 0x7f92140b3e00 2024-04-15T15:17:26.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.465+0000 7f9219dbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9214146de0 con 0x7f92140d0570 2024-04-15T15:17:26.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.472 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51716/0 (socket says 172.21.15.70:51716) 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 -- 172.21.15.70:0/3053476825 learned_addr learned my addr 172.21.15.70:0/3053476825 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 -- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f92140d0570 msgr2=0x7f921414a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f92140d0570 0x7f921414a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 -- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92140b3e00 msgr2=0x7f92141e7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92140b3e00 0x7f92141e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.473 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 -- 172.21.15.70:0/3053476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9214146ae0 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f92137fe640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f920400c780 tx=0x7f920400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9204018070 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92040072d0 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9214147030 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92040047e0 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.466+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92141475f0 con 0x7f921406f6d0 2024-04-15T15:17:26.474 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.467+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92140689e0 con 0x7f921406f6d0 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.468+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f920401b070 con 0x7f921406f6d0 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.468+0000 7f920bfff640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 0x7f91f406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.468+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f920408ae80 con 0x7f921406f6d0 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.468+0000 7f9212ffd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 0x7f91f406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.469+0000 7f9212ffd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 0x7f91f406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f91fc036000 tx=0x7f91fc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.476 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.473+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f921406fab0 con 0x7f921406f6d0 2024-04-15T15:17:26.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.646+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f9214161b60 con 0x7f91f406c980 2024-04-15T15:17:26.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.647+0000 7f920bfff640 1 -- 172.21.15.70:0/3053476825 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9214161b60 con 0x7f91f406c980 2024-04-15T15:17:26.649 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 4 to deep-scrub 2024-04-15T15:17:26.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 msgr2=0x7f91f406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.654 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 0x7f91f406ee40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f91fc036000 tx=0x7f91fc030000 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 msgr2=0x7f92141465a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f920400c780 tx=0x7f920400cc40 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 shutdown_connections 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f92140d0570 0x7f921414a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f92140b3e00 0x7f92141e7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f91f406c980 0x7f91f406ee40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 --2- 172.21.15.70:0/3053476825 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f921406f6d0 0x7f92141465a0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 >> 172.21.15.70:0/3053476825 conn(0x7f921406b9e0 msgr2=0x7f92140aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.655 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 shutdown_connections 2024-04-15T15:17:26.656 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.649+0000 7f9219dbd640 1 -- 172.21.15.70:0/3053476825 wait complete. 2024-04-15T15:17:26.662 INFO:tasks.ceph:Scrubbing osd.5 2024-04-15T15:17:26.662 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:26.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.767+0000 7f9eedda8640 1 Processor -- start 2024-04-15T15:17:26.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.767+0000 7f9eedda8640 1 -- start start 2024-04-15T15:17:26.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9eedda8640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9ee80b67b0 0x7f9ee80aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9eedda8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee8163000 con 0x7f9ee8162210 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9eedda8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ee80adc20 con 0x7f9ee8162670 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9eedda8640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f9ee80ae830 con 0x7f9ee80b67b0 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9ee6ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9ee8162210 0x7f9ee815ec20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49838/0 (socket says 172.21.15.70:49838) 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9ee6ffd640 1 -- 172.21.15.70:0/4156130312 learned_addr learned my addr 172.21.15.70:0/4156130312 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9ee77fe640 1 --2- 172.21.15.70:0/4156130312 >> v2:172.21.15.96:3300/0 conn(0x7f9ee80b67b0 0x7f9ee80aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3051295008 0 0) 0x7f9ee8163000 con 0x7f9ee8162210 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.768+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ed0003520 con 0x7f9ee8162210 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 495993222 0 0) 0x7f9ed0003520 con 0x7f9ee8162210 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 >> v2:172.21.15.96:3300/0 conn(0x7f9ee80b67b0 msgr2=0x7f9ee80aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 --2- 172.21.15.70:0/4156130312 >> v2:172.21.15.96:3300/0 conn(0x7f9ee80b67b0 0x7f9ee80aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 >> v1:172.21.15.96:6789/0 conn(0x7f9ee8162670 legacy=0x7f9ee815f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee80adf00 con 0x7f9ee8162210 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f9edc001670 con 0x7f9ee8162210 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 >> v1:172.21.15.70:6789/0 conn(0x7f9ee8162210 legacy=0x7f9ee815ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed00039b0 0x7f9ed0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 0x7f9ed0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ed00051c0 0x7f9ed000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee80ae830 con 0x7f9ed0004800 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed000c060 con 0x7f9ed00039b0 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9ed000c260 con 0x7f9ed00051c0 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 0x7f9ed0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee77fe640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed00039b0 0x7f9ed0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee7fff640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ed00051c0 0x7f9ed000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 -- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ed00051c0 msgr2=0x7f9ed000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ed00051c0 0x7f9ed000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 -- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed00039b0 msgr2=0x7f9ed0003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed00039b0 0x7f9ed0003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee7fff640 1 -- 172.21.15.70:0/4156130312 reap_dead start 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.769+0000 7f9ee6ffd640 1 -- 172.21.15.70:0/4156130312 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed0003520 con 0x7f9ed0004800 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.770+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 0x7f9ed0004c80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c240 tx=0x7f9ed0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4131cfc055b696f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.770+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc007cf0 con 0x7f9ed0004800 2024-04-15T15:17:26.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.770+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9edc030070 con 0x7f9ed0004800 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.770+0000 7f9ee67fc640 1 -- 172.21.15.70:0/4156130312 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9edc005d70 con 0x7f9ed0004800 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 -- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 msgr2=0x7f9ed0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 0x7f9ed0004c80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00c240 tx=0x7f9ed0005830 comp rx=0 tx=0).stop 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 -- 172.21.15.70:0/4156130312 shutdown_connections 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 --2- 172.21.15.70:0/4156130312 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ed0004800 0x7f9ed0004c80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 -- 172.21.15.70:0/4156130312 >> 172.21.15.70:0/4156130312 conn(0x7f9ee806b9e0 msgr2=0x7f9ee806bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 -- 172.21.15.70:0/4156130312 shutdown_connections 2024-04-15T15:17:26.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.771+0000 7f9eedda8640 1 -- 172.21.15.70:0/4156130312 wait complete. 2024-04-15T15:17:26.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.772+0000 7f9eedda8640 1 Processor -- start 2024-04-15T15:17:26.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.772+0000 7f9eedda8640 1 -- start start 2024-04-15T15:17:26.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed0004800 0x7f9ee80cfd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ee80d0290 0x7f9ee8157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 0x7f9ee81575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee8162ab0 con 0x7f9ee80cf380 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f9ee80ae4c0 con 0x7f9ed0004800 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f9ee80ad840 con 0x7f9ee80d0290 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 0x7f9ee81575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 0x7f9ee81575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51740/0 (socket says 172.21.15.70:51740) 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 -- 172.21.15.70:0/416755101 learned_addr learned my addr 172.21.15.70:0/416755101 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ee80d0290 0x7f9ee8157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 -- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ee80d0290 msgr2=0x7f9ee8157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f9ee80d0290 0x7f9ee8157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee77fe640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed0004800 0x7f9ee80cfd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 -- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed0004800 msgr2=0x7f9ee80cfd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9ed0004800 0x7f9ee80cfd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ee8155d10 con 0x7f9ee80cf380 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ee7fff640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 0x7f9ee81575a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007fd0 tx=0x7f9ee000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0003970 con 0x7f9ee80cf380 2024-04-15T15:17:26.784 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ee0018070 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ee8155ff0 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ee0003d70 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.773+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ee81565b0 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.774+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9ee814acd0 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f9ee00050d0 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9ed406c980 0x7f9ed406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f9ee008c0c0 con 0x7f9ee80cf380 2024-04-15T15:17:26.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ee77fe640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9ed406c980 0x7f9ed406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f9ed4072ec0 0x7f9ed4075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ee00079b0 con 0x7f9ed4072ec0 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.775+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f9ee008c5a0 con 0x7f9ee80cf380 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.776+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f9ed4072ec0 0x7f9ed4075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.776+0000 7f9ee77fe640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9ed406c980 0x7f9ed406ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806a7b0 tx=0x7f9ed8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.776+0000 7f9ee6ffd640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f9ed4072ec0 0x7f9ed4075300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.777+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== osd.5 v2:172.21.15.96:6808/660710685 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9ee00079b0 con 0x7f9ed4072ec0 2024-04-15T15:17:26.791 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.790+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 --> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9ee80689e0 con 0x7f9ed4072ec0 2024-04-15T15:17:26.792 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.790+0000 7f9ecffff640 1 -- 172.21.15.70:0/416755101 <== osd.5 v2:172.21.15.96:6808/660710685 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9ee80689e0 con 0x7f9ed4072ec0 2024-04-15T15:17:26.793 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:26.793 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:26.793 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:26.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.790+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f9ed4072ec0 msgr2=0x7f9ed4075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.794 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.790+0000 7f9eedda8640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.96:6808/660710685,v1:172.21.15.96:6809/660710685] conn(0x7f9ed4072ec0 0x7f9ed4075300 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9ed406c980 msgr2=0x7f9ed406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f9ed406c980 0x7f9ed406ee40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9ee806a7b0 tx=0x7f9ed8008040 comp rx=0 tx=0).stop 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 msgr2=0x7f9ee81575a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 --2- 172.21.15.70:0/416755101 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ee80cf380 0x7f9ee81575a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0007fd0 tx=0x7f9ee000ec10 comp rx=0 tx=0).stop 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9ee7fff640 1 -- 172.21.15.70:0/416755101 reap_dead start 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 shutdown_connections 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 >> 172.21.15.70:0/416755101 conn(0x7f9ee806b9e0 msgr2=0x7f9ee80b58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.795 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 shutdown_connections 2024-04-15T15:17:26.796 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.791+0000 7f9eedda8640 1 -- 172.21.15.70:0/416755101 wait complete. 2024-04-15T15:17:26.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-15T15:17:26.906 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.905+0000 7f0639fbd640 1 Processor -- start 2024-04-15T15:17:26.907 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.905+0000 7f0639fbd640 1 -- start start 2024-04-15T15:17:26.910 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0639fbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f06340d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0639fbd640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06340d0fb0 con 0x7f06340b3160 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0639fbd640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f063415cb50 con 0x7f06340ae280 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0639fbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f063415bda0 con 0x7f06340d0690 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0632ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f06340b3160 0x7f063415b690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49844/0 (socket says 172.21.15.70:49844) 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0632ffd640 1 -- 172.21.15.70:0/1074315345 learned_addr learned my addr 172.21.15.70:0/1074315345 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4249780488 0 0) 0x7f06340d0fb0 con 0x7f06340b3160 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f061c003520 con 0x7f06340b3160 2024-04-15T15:17:26.911 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06337fe640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f06340d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 924130989 0 0) 0x7f061c003520 con 0x7f06340b3160 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 msgr2=0x7f06340d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f06340d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 >> v1:172.21.15.96:6789/0 conn(0x7f06340ae280 legacy=0x7f06340ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f063415bfd0 con 0x7f06340b3160 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 438713127 0 0) 0x7f0628001670 con 0x7f06340b3160 2024-04-15T15:17:26.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 >> v1:172.21.15.70:6789/0 conn(0x7f06340b3160 legacy=0x7f063415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061c0039e0 0x7f061c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c004800 0x7f061c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f061c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f061c00bac0 con 0x7f061c0039e0 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f061c00bc70 con 0x7f061c004800 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f061c00be70 con 0x7f06340d0690 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f06337fe640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061c0039e0 0x7f061c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0632ffd640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c004800 0x7f061c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.913 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.906+0000 7f0633fff640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f061c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 -- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 msgr2=0x7f061c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06340d0690 0x7f061c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 -- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c004800 msgr2=0x7f061c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:26.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c004800 0x7f061c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.914 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 -- 172.21.15.70:0/1074315345 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c003520 con 0x7f061c0039e0 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.907+0000 7f06337fe640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061c0039e0 0x7f061c003e00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f062400ba30 tx=0x7f061c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6460e3ea98a61a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0624036020 con 0x7f061c0039e0 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0624032640 con 0x7f061c0039e0 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f06327fc640 1 -- 172.21.15.70:0/1074315345 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062400b600 con 0x7f061c0039e0 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 -- 172.21.15.70:0/1074315345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061c0039e0 msgr2=0x7f061c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.915 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1074315345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f061c0039e0 0x7f061c003e00 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f062400ba30 tx=0x7f061c005830 comp rx=0 tx=0).stop 2024-04-15T15:17:26.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0633fff640 1 -- 172.21.15.70:0/1074315345 reap_dead start 2024-04-15T15:17:26.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 -- 172.21.15.70:0/1074315345 shutdown_connections 2024-04-15T15:17:26.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 -- 172.21.15.70:0/1074315345 >> 172.21.15.70:0/1074315345 conn(0x7f063406be00 msgr2=0x7f063406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:26.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 -- 172.21.15.70:0/1074315345 shutdown_connections 2024-04-15T15:17:26.916 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.908+0000 7f0639fbd640 1 -- 172.21.15.70:0/1074315345 wait complete. 2024-04-15T15:17:26.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.909+0000 7f0639fbd640 1 Processor -- start 2024-04-15T15:17:26.919 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 -- start start 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c0039e0 0x7f063406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06341526a0 0x7f06340cd840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f063415c870 con 0x7f06340ce160 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f063415c590 con 0x7f061c0039e0 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0639fbd640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f06340b4fb0 con 0x7f06341526a0 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0633fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0633fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51756/0 (socket says 172.21.15.70:51756) 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.910+0000 7f0633fff640 1 -- 172.21.15.70:0/1807623013 learned_addr learned my addr 172.21.15.70:0/1807623013 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0632ffd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06341526a0 0x7f06340cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f06337fe640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c0039e0 0x7f063406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 -- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06341526a0 msgr2=0x7f06340cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06341526a0 0x7f06340cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 -- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c0039e0 msgr2=0x7f063406dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c0039e0 0x7f063406dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:26.921 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 -- 172.21.15.70:0/1807623013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0634152b20 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0633fff640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f06340d11c0 tx=0x7f062c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c01a070 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f062c006d60 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f063406e3f0 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f062c003b90 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.911+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06341e6130 con 0x7f06340ce160 2024-04-15T15:17:26.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.912+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f063406e910 con 0x7f06340ce160 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.912+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f062c01d070 con 0x7f06340ce160 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.913+0000 7f061bfff640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 0x7f062006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.913+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f062c08b020 con 0x7f06340ce160 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.913+0000 7f06337fe640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 0x7f062006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.913+0000 7f06337fe640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 0x7f062006ee70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f062400ba00 tx=0x7f0624030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:26.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:26.917+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f062c059580 con 0x7f06340ce160 2024-04-15T15:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.095+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f06340ceae0 con 0x7f062006c9b0 2024-04-15T15:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.095+0000 7f061bfff640 1 -- 172.21.15.70:0/1807623013 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f06340ceae0 con 0x7f062006c9b0 2024-04-15T15:17:27.097 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 5 to deep-scrub 2024-04-15T15:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 msgr2=0x7f062006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 0x7f062006ee70 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f062400ba00 tx=0x7f0624030000 comp rx=0 tx=0).stop 2024-04-15T15:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 msgr2=0x7f06340ce5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.099 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f06340d11c0 tx=0x7f062c00fd40 comp rx=0 tx=0).stop 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 shutdown_connections 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f062006c9b0 0x7f062006ee70 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f06340ce160 0x7f06340ce5a0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f06341526a0 0x7f06340cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 --2- 172.21.15.70:0/1807623013 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f061c0039e0 0x7f063406dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 >> 172.21.15.70:0/1807623013 conn(0x7f063406be00 msgr2=0x7f06340afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 shutdown_connections 2024-04-15T15:17:27.100 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.097+0000 7f0639fbd640 1 -- 172.21.15.70:0/1807623013 wait complete. 2024-04-15T15:17:27.110 INFO:tasks.ceph:Scrubbing osd.6 2024-04-15T15:17:27.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:27.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.215+0000 7fde78972640 1 Processor -- start 2024-04-15T15:17:27.217 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.215+0000 7fde78972640 1 -- start start 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde78972640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fde7415e8c0 0x7fde740d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde78972640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fde740cfcb0 0x7fde740d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde78972640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde740d2340 con 0x7fde740d0140 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde78972640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fde7415eca0 con 0x7fde7415e8c0 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde78972640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fde740d2620 con 0x7fde740cfcb0 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.216+0000 7fde7259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fde7415e8c0 0x7fde740d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fde7415e8c0 0x7fde740d3100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51764/0 (socket says 172.21.15.70:51764) 2024-04-15T15:17:27.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 -- 172.21.15.70:0/2882411782 learned_addr learned my addr 172.21.15.70:0/2882411782 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:27.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 -- 172.21.15.70:0/2882411782 >> v2:172.21.15.96:3300/0 conn(0x7fde740cfcb0 msgr2=0x7fde740d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:27.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 --2- 172.21.15.70:0/2882411782 >> v2:172.21.15.96:3300/0 conn(0x7fde740cfcb0 0x7fde740d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 -- 172.21.15.70:0/2882411782 >> v1:172.21.15.96:6789/0 conn(0x7fde740d0140 legacy=0x7fde7406c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:27.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 -- 172.21.15.70:0/2882411782 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde740d2900 con 0x7fde7415e8c0 2024-04-15T15:17:27.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.217+0000 7fde7259c640 1 --2- 172.21.15.70:0/2882411782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740d3100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fde740d1e90 tx=0x7fde6402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba85a2144ccb466 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde7159a640 1 -- 172.21.15.70:0/2882411782 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6402c9f0 con 0x7fde7415e8c0 2024-04-15T15:17:27.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde7159a640 1 -- 172.21.15.70:0/2882411782 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6402cb90 con 0x7fde7415e8c0 2024-04-15T15:17:27.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde7159a640 1 -- 172.21.15.70:0/2882411782 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde640343f0 con 0x7fde7415e8c0 2024-04-15T15:17:27.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 -- 172.21.15.70:0/2882411782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 msgr2=0x7fde740d3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 --2- 172.21.15.70:0/2882411782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740d3100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fde740d1e90 tx=0x7fde6402be50 comp rx=0 tx=0).stop 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 -- 172.21.15.70:0/2882411782 shutdown_connections 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 --2- 172.21.15.70:0/2882411782 >> v2:172.21.15.96:3300/0 conn(0x7fde740cfcb0 0x7fde740d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 --2- 172.21.15.70:0/2882411782 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740d3100 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 -- 172.21.15.70:0/2882411782 >> 172.21.15.70:0/2882411782 conn(0x7fde7408bf60 msgr2=0x7fde740d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 -- 172.21.15.70:0/2882411782 shutdown_connections 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.218+0000 7fde78972640 1 -- 172.21.15.70:0/2882411782 wait complete. 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 Processor -- start 2024-04-15T15:17:27.225 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 -- start start 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde740cfcb0 0x7fde74154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde740d0140 0x7fde7415ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fde740d4f60 con 0x7fde7415e8c0 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fde740d4de0 con 0x7fde740d0140 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.220+0000 7fde78972640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fde740d13c0 con 0x7fde740cfcb0 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.226 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51774/0 (socket says 172.21.15.70:51774) 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 -- 172.21.15.70:0/4260230304 learned_addr learned my addr 172.21.15.70:0/4260230304 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde71d9b640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde740d0140 0x7fde7415ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde7259c640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde740cfcb0 0x7fde74154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 -- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde740cfcb0 msgr2=0x7fde74154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fde740cfcb0 0x7fde74154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 -- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde740d0140 msgr2=0x7fde7415ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fde740d0140 0x7fde7415ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde74155340 con 0x7fde7415e8c0 2024-04-15T15:17:27.227 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde72d9d640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740b4c20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c390 tx=0x7fde6c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c017070 con 0x7fde7415e8c0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde6c0035f0 con 0x7fde7415e8c0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde7415f360 con 0x7fde7415e8c0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde6c0039e0 con 0x7fde7415e8c0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.221+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde740ccb80 con 0x7fde7415e8c0 2024-04-15T15:17:27.228 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.222+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fde740689e0 con 0x7fde7415e8c0 2024-04-15T15:17:27.229 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fde6c01a070 con 0x7fde7415e8c0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde5406c980 0x7fde5406ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fde6c08b5c0 con 0x7fde7415e8c0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde7259c640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde5406c980 0x7fde5406ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7fde54072ec0 0x7fde54075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde6c010e50 con 0x7fde54072ec0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.223+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fde6c08baa0 con 0x7fde7415e8c0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.224+0000 7fde71d9b640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7fde54072ec0 0x7fde54075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.230 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.224+0000 7fde7259c640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde5406c980 0x7fde5406ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fde64035000 tx=0x7fde64030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.224+0000 7fde71d9b640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7fde54072ec0 0x7fde54075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.231 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.225+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== osd.6 v2:172.21.15.96:6816/2431924296 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fde6c010e50 con 0x7fde54072ec0 2024-04-15T15:17:27.239 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.238+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 --> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fde740d1ee0 con 0x7fde54072ec0 2024-04-15T15:17:27.240 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.238+0000 7fde62ffd640 1 -- 172.21.15.70:0/4260230304 <== osd.6 v2:172.21.15.96:6816/2431924296 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fde740d1ee0 con 0x7fde54072ec0 2024-04-15T15:17:27.240 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:27.241 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:27.241 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:27.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7fde54072ec0 msgr2=0x7fde54075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.96:6816/2431924296,v1:172.21.15.96:6817/2431924296] conn(0x7fde54072ec0 0x7fde54075300 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde5406c980 msgr2=0x7fde5406ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fde5406c980 0x7fde5406ee40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fde64035000 tx=0x7fde64030000 comp rx=0 tx=0).stop 2024-04-15T15:17:27.242 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 msgr2=0x7fde740b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 --2- 172.21.15.70:0/4260230304 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fde7415e8c0 0x7fde740b4c20 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fde6c00c390 tx=0x7fde6c00c850 comp rx=0 tx=0).stop 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde72d9d640 1 -- 172.21.15.70:0/4260230304 reap_dead start 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 shutdown_connections 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 >> 172.21.15.70:0/4260230304 conn(0x7fde7408bf60 msgr2=0x7fde74074d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 shutdown_connections 2024-04-15T15:17:27.243 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.239+0000 7fde78972640 1 -- 172.21.15.70:0/4260230304 wait complete. 2024-04-15T15:17:27.251 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-15T15:17:27.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.353+0000 7fe98a1f2640 1 Processor -- start 2024-04-15T15:17:27.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.353+0000 7fe98a1f2640 1 -- start start 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe98a1f2640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe9840b5050 0x7fe98406dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe98a1f2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98406e6a0 con 0x7fe984072690 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe98a1f2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe98406e980 con 0x7fe9840d0690 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe98a1f2640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fe98406ec60 con 0x7fe9840b5050 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe982ffd640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fe9840b5050 0x7fe98406dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe9837fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fe984072690 0x7fe98415f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49854/0 (socket says 172.21.15.70:49854) 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe9837fe640 1 -- 172.21.15.70:0/1681762579 learned_addr learned my addr 172.21.15.70:0/1681762579 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe982ffd640 1 -- 172.21.15.70:0/1681762579 >> v1:172.21.15.96:6789/0 conn(0x7fe9840d0690 legacy=0x7fe9841622d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe982ffd640 1 -- 172.21.15.70:0/1681762579 >> v1:172.21.15.70:6789/0 conn(0x7fe984072690 legacy=0x7fe98415f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe982ffd640 1 -- 172.21.15.70:0/1681762579 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98415ee80 con 0x7fe9840b5050 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.354+0000 7fe9827fc640 1 -- 172.21.15.70:0/1681762579 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3668226349 0 0) 0x7fe98406e6a0 con 0x7fe984072690 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.355+0000 7fe982ffd640 1 --2- 172.21.15.70:0/1681762579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe98406dd30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe9780097a0 tx=0x7fe97802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1746d7dbfbc56d server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.355+0000 7fe9827fc640 1 -- 172.21.15.70:0/1681762579 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978034070 con 0x7fe9840b5050 2024-04-15T15:17:27.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.355+0000 7fe9827fc640 1 -- 172.21.15.70:0/1681762579 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe978030610 con 0x7fe9840b5050 2024-04-15T15:17:27.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.355+0000 7fe9827fc640 1 -- 172.21.15.70:0/1681762579 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9780309e0 con 0x7fe9840b5050 2024-04-15T15:17:27.365 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1681762579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 msgr2=0x7fe98406dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1681762579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe98406dd30 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fe9780097a0 tx=0x7fe97802b960 comp rx=0 tx=0).stop 2024-04-15T15:17:27.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1681762579 shutdown_connections 2024-04-15T15:17:27.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1681762579 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe98406dd30 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.366 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1681762579 >> 172.21.15.70:0/1681762579 conn(0x7fe98406be00 msgr2=0x7fe98406c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1681762579 shutdown_connections 2024-04-15T15:17:27.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.356+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1681762579 wait complete. 2024-04-15T15:17:27.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.357+0000 7fe98a1f2640 1 Processor -- start 2024-04-15T15:17:27.367 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.357+0000 7fe98a1f2640 1 -- start start 2024-04-15T15:17:27.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe984072690 0x7fe98414a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe9840d0690 0x7fe9840b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.368 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe98406e3f0 con 0x7fe9840b5050 2024-04-15T15:17:27.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fe98406e270 con 0x7fe984072690 2024-04-15T15:17:27.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe98a1f2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fe9841629e0 con 0x7fe9840d0690 2024-04-15T15:17:27.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.369 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51784/0 (socket says 172.21.15.70:51784) 2024-04-15T15:17:27.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 -- 172.21.15.70:0/1824575945 learned_addr learned my addr 172.21.15.70:0/1824575945 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:27.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe9837fe640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe984072690 0x7fe98414a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe983fff640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe9840d0690 0x7fe9840b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.370 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 -- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe9840d0690 msgr2=0x7fe9840b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe9840d0690 0x7fe9840b3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 -- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe984072690 msgr2=0x7fe98414a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe984072690 0x7fe98414a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.371 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 -- 172.21.15.70:0/1824575945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe98409b4d0 con 0x7fe9840b5050 2024-04-15T15:17:27.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.358+0000 7fe982ffd640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe9780098c0 tx=0x7fe9780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.359+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978034040 con 0x7fe9840b5050 2024-04-15T15:17:27.372 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.359+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe978039a10 con 0x7fe9840b5050 2024-04-15T15:17:27.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.359+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe98409ac30 con 0x7fe9840b5050 2024-04-15T15:17:27.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.359+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe978039de0 con 0x7fe9840b5050 2024-04-15T15:17:27.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.359+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe98409b1f0 con 0x7fe9840b5050 2024-04-15T15:17:27.373 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.360+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9840689e0 con 0x7fe9840b5050 2024-04-15T15:17:27.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.360+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fe978037070 con 0x7fe9840b5050 2024-04-15T15:17:27.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.361+0000 7fe96bfff640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 0x7fe97006ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.361+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fe9780add20 con 0x7fe9840b5050 2024-04-15T15:17:27.374 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.361+0000 7fe9837fe640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 0x7fe97006ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.361+0000 7fe9837fe640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 0x7fe97006ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f60 tx=0x7fe974007b90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.375 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.365+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fe97807c170 con 0x7fe9840b5050 2024-04-15T15:17:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.544+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7fe98406f9e0 con 0x7fe97006c980 2024-04-15T15:17:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.544+0000 7fe96bfff640 1 -- 172.21.15.70:0/1824575945 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe98406f9e0 con 0x7fe97006c980 2024-04-15T15:17:27.546 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 6 to deep-scrub 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 msgr2=0x7fe97006ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 0x7fe97006ee40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe974000f60 tx=0x7fe974007b90 comp rx=0 tx=0).stop 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 msgr2=0x7fe9840b3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe9780098c0 tx=0x7fe9780046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 shutdown_connections 2024-04-15T15:17:27.548 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fe9840d0690 0x7fe9840b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fe97006c980 0x7fe97006ee40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe9840b5050 0x7fe9840b3730 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 --2- 172.21.15.70:0/1824575945 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fe984072690 0x7fe98414a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 >> 172.21.15.70:0/1824575945 conn(0x7fe98406be00 msgr2=0x7fe984158680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 shutdown_connections 2024-04-15T15:17:27.549 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.546+0000 7fe98a1f2640 1 -- 172.21.15.70:0/1824575945 wait complete. 2024-04-15T15:17:27.559 INFO:tasks.ceph:Scrubbing osd.7 2024-04-15T15:17:27.559 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-15T15:17:27.665 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.664+0000 7fa26d511640 1 Processor -- start 2024-04-15T15:17:27.666 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.664+0000 7fa26d511640 1 -- start start 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa26d511640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2680d1590 0x7fa26806c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa26d511640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2680d03c0 con 0x7fa26815e8c0 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa26d511640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2680d2340 con 0x7fa2680d3100 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa26d511640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7fa26815eca0 con 0x7fa2680d1590 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa2677fe640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7fa2680d1590 0x7fa26806c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.667 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa266ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa26815e8c0 0x7fa2680cfcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49862/0 (socket says 172.21.15.70:49862) 2024-04-15T15:17:27.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa266ffd640 1 -- 172.21.15.70:0/4078084990 learned_addr learned my addr 172.21.15.70:0/4078084990 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:27.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa2677fe640 1 -- 172.21.15.70:0/4078084990 >> v1:172.21.15.96:6789/0 conn(0x7fa2680d3100 legacy=0x7fa2680d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:27.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa2677fe640 1 -- 172.21.15.70:0/4078084990 >> v1:172.21.15.70:6789/0 conn(0x7fa26815e8c0 legacy=0x7fa2680cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.668 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.665+0000 7fa2677fe640 1 -- 172.21.15.70:0/4078084990 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2680d2620 con 0x7fa2680d1590 2024-04-15T15:17:27.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa2677fe640 1 --2- 172.21.15.70:0/4078084990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d1590 0x7fa26806c970 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa260009d90 tx=0x7fa26002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e957993146ba1ca6 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.669 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa2667fc640 1 -- 172.21.15.70:0/4078084990 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260039070 con 0x7fa2680d1590 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa2667fc640 1 -- 172.21.15.70:0/4078084990 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa26002cc60 con 0x7fa2680d1590 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa2667fc640 1 -- 172.21.15.70:0/4078084990 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260034660 con 0x7fa2680d1590 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 -- 172.21.15.70:0/4078084990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d1590 msgr2=0x7fa26806c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 --2- 172.21.15.70:0/4078084990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d1590 0x7fa26806c970 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa260009d90 tx=0x7fa26002beb0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 -- 172.21.15.70:0/4078084990 shutdown_connections 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 --2- 172.21.15.70:0/4078084990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d1590 0x7fa26806c970 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.670 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 -- 172.21.15.70:0/4078084990 >> 172.21.15.70:0/4078084990 conn(0x7fa26808bf60 msgr2=0x7fa2680d45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 -- 172.21.15.70:0/4078084990 shutdown_connections 2024-04-15T15:17:27.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.666+0000 7fa26d511640 1 -- 172.21.15.70:0/4078084990 wait complete. 2024-04-15T15:17:27.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.668+0000 7fa26d511640 1 Processor -- start 2024-04-15T15:17:27.671 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.668+0000 7fa26d511640 1 -- start start 2024-04-15T15:17:27.674 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.668+0000 7fa26d511640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2680d1590 0x7fa2680cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 0x7fa2680cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.677 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa26815e8c0 0x7fa2680cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa2680d2be0 con 0x7fa2680d3100 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fa2680d4de0 con 0x7fa26815e8c0 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fa2680d4f60 con 0x7fa2680d1590 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 0x7fa2680cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 0x7fa2680cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51806/0 (socket says 172.21.15.70:51806) 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 -- 172.21.15.70:0/1017046173 learned_addr learned my addr 172.21.15.70:0/1017046173 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 -- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2680d1590 msgr2=0x7fa2680cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.678 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fa2680d1590 0x7fa2680cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 -- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa26815e8c0 msgr2=0x7fa2680cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa2677fe640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa26815e8c0 0x7fa2680cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa26815e8c0 0x7fa2680cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2680ce500 con 0x7fa2680d3100 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa2677fe640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fa26815e8c0 0x7fa2680cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25e7fc640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 0x7fa2680cd980 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250011790 con 0x7fa2680d3100 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa250017070 con 0x7fa2680d3100 2024-04-15T15:17:27.679 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2680ce7e0 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa250011bb0 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.669+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2680c9240 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.670+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa2680689e0 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fa25001c460 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa24806c980 0x7fa24806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fa25008b9b0 con 0x7fa2680d3100 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa266ffd640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa24806c980 0x7fa24806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.680 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7fa248072ec0 0x7fa248075300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa25000ae50 con 0x7fa248072ec0 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.671+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7fa25008de60 con 0x7fa2680d3100 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.672+0000 7fa266ffd640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa24806c980 0x7fa24806ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa258000f60 tx=0x7fa258007a80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.672+0000 7fa2677fe640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7fa248072ec0 0x7fa248075300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.672+0000 7fa2677fe640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7fa248072ec0 0x7fa248075300 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.681 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.673+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== osd.7 v2:172.21.15.96:6824/1112579369 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa25000ae50 con 0x7fa248072ec0 2024-04-15T15:17:27.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.686+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 --> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa2680d1ee0 con 0x7fa248072ec0 2024-04-15T15:17:27.688 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa25ffff640 1 -- 172.21.15.70:0/1017046173 <== osd.7 v2:172.21.15.96:6824/1112579369 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa2680d1ee0 con 0x7fa248072ec0 2024-04-15T15:17:27.689 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-04-15T15:17:27.689 INFO:teuthology.orchestra.run.smithi070.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-15T15:17:27.689 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-04-15T15:17:27.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7fa248072ec0 msgr2=0x7fa248075300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.691 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.96:6824/1112579369,v1:172.21.15.96:6825/1112579369] conn(0x7fa248072ec0 0x7fa248075300 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa24806c980 msgr2=0x7fa24806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fa24806c980 0x7fa24806ee40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa258000f60 tx=0x7fa258007a80 comp rx=0 tx=0).stop 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 msgr2=0x7fa2680cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 --2- 172.21.15.70:0/1017046173 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa2680d3100 0x7fa2680cd980 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa25000c430 tx=0x7fa25000c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa2677fe640 1 -- 172.21.15.70:0/1017046173 reap_dead start 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 shutdown_connections 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 >> 172.21.15.70:0/1017046173 conn(0x7fa26808bf60 msgr2=0x7fa268074e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.692 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 shutdown_connections 2024-04-15T15:17:27.693 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.687+0000 7fa26d511640 1 -- 172.21.15.70:0/1017046173 wait complete. 2024-04-15T15:17:27.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-15T15:17:27.803 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.802+0000 7f58da64a640 1 Processor -- start 2024-04-15T15:17:27.804 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.802+0000 7f58da64a640 1 -- start start 2024-04-15T15:17:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.802+0000 7f58da64a640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f58d40b0560 0x7f58d406b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58da64a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58d40cbd10 con 0x7f58d415eb00 2024-04-15T15:17:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58da64a640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58d40cbff0 con 0x7f58d40b09a0 2024-04-15T15:17:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f58d415eb00 0x7f58d40ae2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:49874/0 (socket says 172.21.15.70:49874) 2024-04-15T15:17:27.805 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f58d415eee0 con 0x7f58d40b0560 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d3fff640 1 -- 172.21.15.70:0/3773759664 learned_addr learned my addr 172.21.15.70:0/3773759664 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d37fe640 1 --2- 172.21.15.70:0/3773759664 >> v2:172.21.15.96:3300/0 conn(0x7f58d40b0560 0x7f58d406b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444559595 0 0) 0x7f58d40cbd10 con 0x7f58d415eb00 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58bc003520 con 0x7f58d415eb00 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d37fe640 1 -- 172.21.15.70:0/3773759664 >> v1:172.21.15.96:6789/0 conn(0x7f58d40b09a0 legacy=0x7f58d40ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2001368387 0 0) 0x7f58bc003520 con 0x7f58d415eb00 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d37fe640 1 -- 172.21.15.70:0/3773759664 >> v1:172.21.15.70:6789/0 conn(0x7f58d415eb00 legacy=0x7f58d40ae2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.806 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.803+0000 7f58d37fe640 1 -- 172.21.15.70:0/3773759664 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d40cc2d0 con 0x7f58d40b0560 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.804+0000 7f58d37fe640 1 --2- 172.21.15.70:0/3773759664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b0560 0x7f58d406b9e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f58c80097a0 tx=0x7f58c802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29066f35a6c53358 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.804+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c8038070 con 0x7f58d40b0560 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.804+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58c8031500 con 0x7f58d40b0560 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.804+0000 7f58d2ffd640 1 -- 172.21.15.70:0/3773759664 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58c4001670 con 0x7f58d40b0560 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b0560 msgr2=0x7f58d406b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.808 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 --2- 172.21.15.70:0/3773759664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b0560 0x7f58d406b9e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f58c80097a0 tx=0x7f58c802b820 comp rx=0 tx=0).stop 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 shutdown_connections 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 --2- 172.21.15.70:0/3773759664 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b0560 0x7f58d406b9e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 >> 172.21.15.70:0/3773759664 conn(0x7f58d409a9d0 msgr2=0x7f58d40d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 shutdown_connections 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.805+0000 7f58da64a640 1 -- 172.21.15.70:0/3773759664 wait complete. 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.806+0000 7f58da64a640 1 Processor -- start 2024-04-15T15:17:27.809 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.806+0000 7f58da64a640 1 -- start start 2024-04-15T15:17:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f58d40b0560 0x7f58d41f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b09a0 0x7f58d409eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f58d40cec60 con 0x7f58d415eb00 2024-04-15T15:17:27.812 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f58d40cc890 con 0x7f58d40b09a0 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58da64a640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f58d40cede0 con 0x7f58d40b0560 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51812/0 (socket says 172.21.15.70:51812) 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 -- 172.21.15.70:0/3680046476 learned_addr learned my addr 172.21.15.70:0/3680046476 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d37fe640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b09a0 0x7f58d409eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d3fff640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f58d40b0560 0x7f58d41f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 -- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f58d40b0560 msgr2=0x7f58d41f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.813 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f58d40b0560 0x7f58d41f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 -- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b09a0 msgr2=0x7f58d409eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b09a0 0x7f58d409eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.807+0000 7f58d8be7640 1 -- 172.21.15.70:0/3680046476 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58d40b4570 con 0x7f58d415eb00 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58d8be7640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00c7a0 tx=0x7f58cc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc003b70 con 0x7f58d415eb00 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58cc003d30 con 0x7f58d415eb00 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58cc00aa90 con 0x7f58d415eb00 2024-04-15T15:17:27.814 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58d40b4850 con 0x7f58d415eb00 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.808+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58d40b4e10 con 0x7f58d415eb00 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.809+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58d40b5050 con 0x7f58d415eb00 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.809+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f58cc00ac30 con 0x7f58d415eb00 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.810+0000 7f58d0ff9640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 0x7f58c006ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.810+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f58cc08b8d0 con 0x7f58d415eb00 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.810+0000 7f58d3fff640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 0x7f58c006ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:27.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.810+0000 7f58d3fff640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 0x7f58c006ee70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005880 tx=0x7f58c40057f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:27.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.814+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f58d40b5050 con 0x7f58d415eb00 2024-04-15T15:17:27.990 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.988+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f58d40690f0 con 0x7f58c006c9b0 2024-04-15T15:17:27.991 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.989+0000 7f58d0ff9640 1 -- 172.21.15.70:0/3680046476 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f58d40690f0 con 0x7f58c006c9b0 2024-04-15T15:17:27.991 INFO:teuthology.orchestra.run.smithi070.stderr:instructed osd(s) 7 to deep-scrub 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 msgr2=0x7f58c006ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 0x7f58c006ee70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f58c4005880 tx=0x7f58c40057f0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 msgr2=0x7f58d40c71c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00c7a0 tx=0x7f58cc00cc60 comp rx=0 tx=0).stop 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 shutdown_connections 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f58c006c9b0 0x7f58c006ee70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.993 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f58d415eb00 0x7f58d40c71c0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f58d40b09a0 0x7f58d409eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 --2- 172.21.15.70:0/3680046476 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f58d40b0560 0x7f58d41f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:27.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 >> 172.21.15.70:0/3680046476 conn(0x7f58d409a9d0 msgr2=0x7f58d40d1560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:27.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 shutdown_connections 2024-04-15T15:17:27.994 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:27.991+0000 7f58da64a640 1 -- 172.21.15.70:0/3680046476 wait complete. 2024-04-15T15:17:28.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:28.108 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.107+0000 7fca41be2640 1 Processor -- start 2024-04-15T15:17:28.109 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.107+0000 7fca41be2640 1 -- start start 2024-04-15T15:17:28.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca41be2640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.110 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca41be2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca41be2640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca3c0d0fb0 con 0x7fca3c15ef70 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca41be2640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7fca3c15cb50 con 0x7fca3c0d0690 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca41be2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c15bfd0 con 0x7fca3c0b3160 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca3b7fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:58928/0 (socket says 172.21.15.70:58928) 2024-04-15T15:17:28.111 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca3bfff640 1 --1- >> v1:172.21.15.96:6789/0 conn(0x7fca3c15ef70 0x7fca3c15f3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.96:6789/0 says I am v1:172.21.15.70:33414/0 (socket says 172.21.15.70:33414) 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca3b7fe640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3300/0 says I am v2:172.21.15.70:33386/0 (socket says 172.21.15.70:33386) 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 -- 172.21.15.70:0/4084136971 learned_addr learned my addr 172.21.15.70:0/4084136971 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 -- 172.21.15.70:0/4084136971 >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 msgr2=0x7fca3c0d0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 --2- 172.21.15.70:0/4084136971 >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 -- 172.21.15.70:0/4084136971 >> v1:172.21.15.96:6789/0 conn(0x7fca3c15ef70 legacy=0x7fca3c15f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca33fff640 1 -- 172.21.15.70:0/4084136971 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c15c2b0 con 0x7fca3c0b3160 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca3b7fe640 1 --2- 172.21.15.70:0/4084136971 >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:28.112 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.108+0000 7fca3affd640 1 -- 172.21.15.70:0/4084136971 <== mon.1 v1:172.21.15.96:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922387086 0 0) 0x7fca3c0d0fb0 con 0x7fca3c15ef70 2024-04-15T15:17:28.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.109+0000 7fca33fff640 1 --2- 172.21.15.70:0/4084136971 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fca24009870 tx=0x7fca2402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1819ee22bc5b197e server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.109+0000 7fca3affd640 1 -- 172.21.15.70:0/4084136971 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24034070 con 0x7fca3c0b3160 2024-04-15T15:17:28.116 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.109+0000 7fca3affd640 1 -- 172.21.15.70:0/4084136971 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca24030840 con 0x7fca3c0b3160 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.109+0000 7fca3affd640 1 -- 172.21.15.70:0/4084136971 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24030c10 con 0x7fca3c0b3160 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 -- 172.21.15.70:0/4084136971 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 msgr2=0x7fca3c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 --2- 172.21.15.70:0/4084136971 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fca24009870 tx=0x7fca2402b980 comp rx=0 tx=0).stop 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 -- 172.21.15.70:0/4084136971 shutdown_connections 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 --2- 172.21.15.70:0/4084136971 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c0b3160 0x7fca3c15b690 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 --2- 172.21.15.70:0/4084136971 >> v2:172.21.15.96:3300/0 conn(0x7fca3c0d0690 0x7fca3c0d0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 -- 172.21.15.70:0/4084136971 >> 172.21.15.70:0/4084136971 conn(0x7fca3c06be00 msgr2=0x7fca3c06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.117 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 -- 172.21.15.70:0/4084136971 shutdown_connections 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.110+0000 7fca41be2640 1 -- 172.21.15.70:0/4084136971 wait complete. 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.111+0000 7fca41be2640 1 Processor -- start 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.111+0000 7fca41be2640 1 -- start start 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca3c0b3160 0x7fca3c14a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 0x7fca3c09d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c0b4fb0 con 0x7fca3c0d0690 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7fca3c15c870 con 0x7fca3c0b3160 2024-04-15T15:17:28.118 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca41be2640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7fca3c15bbd0 con 0x7fca3c15ef70 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:51828/0 (socket says 172.21.15.70:51828) 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 -- 172.21.15.70:0/3423703364 learned_addr learned my addr 172.21.15.70:0/3423703364 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca3b7fe640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca3c0b3160 0x7fca3c14a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca3bfff640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 0x7fca3c09d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 -- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 msgr2=0x7fca3c09d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 0x7fca3c09d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.119 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 -- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca3c0b3160 msgr2=0x7fca3c14a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca3c0b3160 0x7fca3c14a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 -- 172.21.15.70:0/3423703364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca3c159bb0 con 0x7fca3c0d0690 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca3bfff640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 0x7fca3c09d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.112+0000 7fca33fff640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.113+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24034040 con 0x7fca3c0d0690 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.113+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca24039d90 con 0x7fca3c0d0690 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.113+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca3c09d800 con 0x7fca3c0d0690 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.113+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca24043630 con 0x7fca3c0d0690 2024-04-15T15:17:28.120 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.113+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca3c09ddc0 con 0x7fca3c0d0690 2024-04-15T15:17:28.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.114+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca3c0c5180 con 0x7fca3c0d0690 2024-04-15T15:17:28.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.114+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7fca2404a060 con 0x7fca3c0d0690 2024-04-15T15:17:28.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.114+0000 7fca38ff9640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 0x7fca1c06ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:28.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.114+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7fca240ada60 con 0x7fca3c0d0690 2024-04-15T15:17:28.121 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.115+0000 7fca3b7fe640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 0x7fca1c06ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.115+0000 7fca3b7fe640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 0x7fca1c06ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009560 tx=0x7fca2c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:28.122 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.119+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7fca2403e070 con 0x7fca3c0d0690 2024-04-15T15:17:28.294 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.292+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca3c0689e0 con 0x7fca1c06c980 2024-04-15T15:17:28.295 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.293+0000 7fca38ff9640 1 -- 172.21.15.70:0/3423703364 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13385 (secure 0 0 0) 0x7fca3c0689e0 con 0x7fca1c06c980 2024-04-15T15:17:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:28.295 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:17:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.294+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 msgr2=0x7fca1c06ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 0x7fca1c06ee40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009560 tx=0x7fca2c009190 comp rx=0 tx=0).stop 2024-04-15T15:17:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 msgr2=0x7fca3c09cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fca24000c00 tx=0x7fca240046c0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.297 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 shutdown_connections 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7fca3c15ef70 0x7fca3c09d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7fca1c06c980 0x7fca1c06ee40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca3c0d0690 0x7fca3c09cc80 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 --2- 172.21.15.70:0/3423703364 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7fca3c0b3160 0x7fca3c14a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 >> 172.21.15.70:0/3423703364 conn(0x7fca3c06be00 msgr2=0x7fca3c15e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 shutdown_connections 2024-04-15T15:17:28.298 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:28.295+0000 7fca41be2640 1 -- 172.21.15.70:0/3423703364 wait complete. 2024-04-15T15:17:28.307 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-04-15T15:17:28.096930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238128,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736592,"statfs":{"total":773094113280,"available":772850270208,"internally_reserved":0,"allocated":5447680,"data_stored":3042944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-133,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-1,"num_write_kb":-1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001621"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":216,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:17:20.859671+0000","last_change":"2024-04-15T15:14:59.274158+0000","last_active":"2024-04-15T15:17:20.859671+0000","last_peered":"2024-04-15T15:17:20.859671+0000","last_clean":"2024-04-15T15:17:20.859671+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:17:20.859671+0000","last_undegraded":"2024-04-15T15:17:20.859671+0000","last_fullsized":"2024-04-15T15:17:20.859671+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T15:14:58.236879+0000","last_clean_scrub_stamp":"2024-04-15T15:14:58.236879+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T02:49:15.428711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28120,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343720,"statfs":{"total":96636764160,"available":96607969280,"internally_reserved":0,"allocated":1126400,"data_stored":823144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32280,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339560,"statfs":{"total":96636764160,"available":96603709440,"internally_reserved":0,"allocated":1126400,"data_stored":823144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:28.310 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-15T15:14:58.236879+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=14, tm_sec=58, tm_wday=0, tm_yday=106, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=15, tm_hour=15, tm_min=17, tm_sec=23, tm_wday=0, tm_yday=106, tm_isdst=0) 2024-04-15T15:17:28.310 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-15T15:17:41.926 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:41.929 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:48.312 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-15T15:17:48.427 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.425+0000 7f978cceb640 1 Processor -- start 2024-04-15T15:17:48.428 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.425+0000 7f978cceb640 1 -- start start 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.426+0000 7f978cceb640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 0x7f9788071d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.426+0000 7f978cceb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.426+0000 7f978cceb640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f978815db40 con 0x7f97880af3e0 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.426+0000 7f978cceb640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f978815cfc0 con 0x7f9788072d80 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.426+0000 7f978cceb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f978815d2a0 con 0x7f97880d0da0 2024-04-15T15:17:48.430 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.96:3301/0 says I am v2:172.21.15.70:46104/0 (socket says 172.21.15.70:46104) 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 -- 172.21.15.70:0/2681571403 learned_addr learned my addr 172.21.15.70:0/2681571403 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f978659c640 1 --2- 172.21.15.70:0/2681571403 >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 0x7f9788071d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 -- 172.21.15.70:0/2681571403 >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 msgr2=0x7f9788071d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 --2- 172.21.15.70:0/2681571403 >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 0x7f9788071d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 -- 172.21.15.70:0/2681571403 >> v1:172.21.15.96:6789/0 conn(0x7f97880af3e0 legacy=0x7f97880d0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.431 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f9786d9d640 1 -- 172.21.15.70:0/2681571403 --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978815d580 con 0x7f97880d0da0 2024-04-15T15:17:48.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.427+0000 7f978659c640 1 --2- 172.21.15.70:0/2681571403 >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 0x7f9788071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:48.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.428+0000 7f9786d9d640 1 --2- 172.21.15.70:0/2681571403 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9780009e50 tx=0x7f978002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec127e54b4018636 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.428+0000 7f978559a640 1 -- 172.21.15.70:0/2681571403 <== mon.2 v2:172.21.15.96:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f978002ca70 con 0x7f97880d0da0 2024-04-15T15:17:48.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.428+0000 7f978559a640 1 -- 172.21.15.70:0/2681571403 <== mon.2 v2:172.21.15.96:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f978002cc10 con 0x7f97880d0da0 2024-04-15T15:17:48.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.428+0000 7f978559a640 1 -- 172.21.15.70:0/2681571403 <== mon.2 v2:172.21.15.96:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97800343f0 con 0x7f97880d0da0 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 -- 172.21.15.70:0/2681571403 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 msgr2=0x7f97880af840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 --2- 172.21.15.70:0/2681571403 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9780009e50 tx=0x7f978002bed0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 -- 172.21.15.70:0/2681571403 shutdown_connections 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 --2- 172.21.15.70:0/2681571403 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f97880af840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 --2- 172.21.15.70:0/2681571403 >> v2:172.21.15.96:3300/0 conn(0x7f9788072d80 0x7f9788071d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 -- 172.21.15.70:0/2681571403 >> 172.21.15.70:0/2681571403 conn(0x7f978806be00 msgr2=0x7f978806c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 -- 172.21.15.70:0/2681571403 shutdown_connections 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.429+0000 7f978cceb640 1 -- 172.21.15.70:0/2681571403 wait complete. 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.430+0000 7f978cceb640 1 Processor -- start 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.430+0000 7f978cceb640 1 -- start start 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9788072d80 0x7f9788157250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f9788153590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f978815cd10 con 0x7f97880af3e0 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f978815cb90 con 0x7f9788072d80 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978cceb640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f978806e060 con 0x7f97880d0da0 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53792/0 (socket says 172.21.15.70:53792) 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 -- 172.21.15.70:0/3620628064 learned_addr learned my addr 172.21.15.70:0/3620628064 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:48.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9786d9d640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f9788153590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f978659c640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9788072d80 0x7f9788157250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 -- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 msgr2=0x7f9788153590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f9788153590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 -- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9788072d80 msgr2=0x7f9788157250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9788072d80 0x7f9788157250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.431+0000 7f9785d9b640 1 -- 172.21.15.70:0/3620628064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97881527d0 con 0x7f97880af3e0 2024-04-15T15:17:48.439 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f9785d9b640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c430 tx=0x7f977c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c0052d0 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f977c005490 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9788152ab0 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f977c007260 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.432+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9788151080 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.433+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97880689e0 con 0x7f97880af3e0 2024-04-15T15:17:48.440 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.433+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f977c007400 con 0x7f97880af3e0 2024-04-15T15:17:48.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.434+0000 7f9776ffd640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 0x7f976806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.434+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f977c08af20 con 0x7f97880af3e0 2024-04-15T15:17:48.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.434+0000 7f978659c640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 0x7f976806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.434+0000 7f978659c640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 0x7f976806ee40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9778009560 tx=0x7f9778009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.441 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.438+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f977c058e30 con 0x7f97880af3e0 2024-04-15T15:17:48.610 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.609+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 --> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f97880ce150 con 0x7f976806c980 2024-04-15T15:17:48.611 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.609+0000 7f9776ffd640 1 -- 172.21.15.70:0/3620628064 <== mgr.4105 v2:172.21.15.70:6832/2172332499 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7f97880ce150 con 0x7f976806c980 2024-04-15T15:17:48.611 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:48.612 INFO:teuthology.orchestra.run.smithi070.stderr:dumped all 2024-04-15T15:17:48.613 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 msgr2=0x7f976806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 0x7f976806ee40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f9778009560 tx=0x7f9778009190 comp rx=0 tx=0).stop 2024-04-15T15:17:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 msgr2=0x7f97880cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f977c00c430 tx=0x7f977c00c8f0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 shutdown_connections 2024-04-15T15:17:48.614 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.611+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f97880d0da0 0x7f9788153590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f976806c980 0x7f976806ee40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97880af3e0 0x7f97880cd690 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 --2- 172.21.15.70:0/3620628064 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f9788072d80 0x7f9788157250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 >> 172.21.15.70:0/3620628064 conn(0x7f978806be00 msgr2=0x7f97880b13c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 shutdown_connections 2024-04-15T15:17:48.615 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.612+0000 7f978cceb640 1 -- 172.21.15.70:0/3620628064 wait complete. 2024-04-15T15:17:48.624 INFO:teuthology.orchestra.run.smithi070.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-04-15T15:17:48.101220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":238120,"kb_used_data":5320,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754736600,"statfs":{"total":773094113280,"available":772850278400,"internally_reserved":0,"allocated":5447680,"data_stored":3042944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":238276145},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002580"},"pg_stats":[{"pgid":"1.0","version":"45'119","reported_seq":227,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-04-15T15:17:28.029715+0000","last_change":"2024-04-15T15:17:28.029747+0000","last_active":"2024-04-15T15:17:28.029715+0000","last_peered":"2024-04-15T15:17:28.029715+0000","last_clean":"2024-04-15T15:17:28.029715+0000","last_became_active":"2024-04-15T15:14:59.273289+0000","last_became_peered":"2024-04-15T15:14:59.273289+0000","last_unstale":"2024-04-15T15:17:28.029715+0000","last_undegraded":"2024-04-15T15:17:28.029715+0000","last_fullsized":"2024-04-15T15:17:28.029715+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"45'119","last_scrub_stamp":"2024-04-15T15:17:28.029641+0000","last_deep_scrub":"45'119","last_deep_scrub_stamp":"2024-04-15T15:17:28.029641+0000","last_clean_scrub_stamp":"2024-04-15T15:17:28.029641+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-16T20:09:15.671379+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542181,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28116,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343724,"statfs":{"total":96636764160,"available":96607973376,"internally_reserved":0,"allocated":1126400,"data_stored":823144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27544,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27540,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31704,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340136,"statfs":{"total":96636764160,"available":96604299264,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574886,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":532480,"data_stored":232776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32276,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339564,"statfs":{"total":96636764160,"available":96603713536,"internally_reserved":0,"allocated":1126400,"data_stored":823144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T15:17:48.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-15T15:17:48.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.727+0000 7f4ad1ace640 1 Processor -- start 2024-04-15T15:17:48.729 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.727+0000 7f4ad1ace640 1 -- start start 2024-04-15T15:17:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4ad1ace640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc072720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4ad1ace640 1 --2- >> v2:172.21.15.96:3300/0 conn(0x7f4acc072c60 0x7f4acc06dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4ad1ace640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4acc0b4190 con 0x7f4acc158770 2024-04-15T15:17:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4ad1ace640 1 -- --> v2:172.21.15.96:3300/0 -- mon_getmap magic: 0 -- 0x7f4acc0b4da0 con 0x7f4acc072c60 2024-04-15T15:17:48.731 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4ad1ace640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc0b4470 con 0x7f4acc0b5230 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f4acc158770 0x7f4acc15bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44276/0 (socket says 172.21.15.70:44276) 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acb7fe640 1 -- 172.21.15.70:0/4200605090 learned_addr learned my addr 172.21.15.70:0/4200605090 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acaffd640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc072720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acbfff640 1 --2- 172.21.15.70:0/4200605090 >> v2:172.21.15.96:3300/0 conn(0x7f4acc072c60 0x7f4acc06dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1975288100 0 0) 0x7f4acc0b4190 con 0x7f4acc158770 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4ab4003520 con 0x7f4acc158770 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 <== mon.0 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1072413829 0 0) 0x7f4ab4003520 con 0x7f4acc158770 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acbfff640 1 -- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 msgr2=0x7f4acc072720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:48.732 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acbfff640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc072720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acbfff640 1 -- 172.21.15.70:0/4200605090 >> v1:172.21.15.70:6789/0 conn(0x7f4acc158770 legacy=0x7f4acc15bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acbfff640 1 -- 172.21.15.70:0/4200605090 --> v2:172.21.15.96:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc0b4750 con 0x7f4acc072c60 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.728+0000 7f4acaffd640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.729+0000 7f4acbfff640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc072c60 0x7f4acc06dd30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009d90 tx=0x7f4ac402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7597e203b6912206 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.729+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 <== mon.1 v2:172.21.15.96:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4039070 con 0x7f4acc072c60 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.729+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 <== mon.1 v2:172.21.15.96:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ac402c8a0 con 0x7f4acc072c60 2024-04-15T15:17:48.733 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.729+0000 7f4aca7fc640 1 -- 172.21.15.70:0/4200605090 <== mon.1 v2:172.21.15.96:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc001670 con 0x7f4acc072c60 2024-04-15T15:17:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 -- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc072c60 msgr2=0x7f4acc06dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc072c60 0x7f4acc06dd30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4009d90 tx=0x7f4ac402bd70 comp rx=0 tx=0).stop 2024-04-15T15:17:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 -- 172.21.15.70:0/4200605090 shutdown_connections 2024-04-15T15:17:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc072c60 0x7f4acc06dd30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/4200605090 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc072720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 -- 172.21.15.70:0/4200605090 >> 172.21.15.70:0/4200605090 conn(0x7f4acc06be00 msgr2=0x7f4acc06c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 -- 172.21.15.70:0/4200605090 shutdown_connections 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.730+0000 7f4ad1ace640 1 -- 172.21.15.70:0/4200605090 wait complete. 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.731+0000 7f4ad1ace640 1 Processor -- start 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.731+0000 7f4ad1ace640 1 -- start start 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.738 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc158770 0x7f4acc151100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc158b50 con 0x7f4acc072c60 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc071fa0 con 0x7f4acc158770 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4ad1ace640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f4acc0b3d20 con 0x7f4acc0b5230 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53798/0 (socket says 172.21.15.70:53798) 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 -- 172.21.15.70:0/2527009688 learned_addr learned my addr 172.21.15.70:0/2527009688 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acaffd640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc150bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.739 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acbfff640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc158770 0x7f4acc151100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 -- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 msgr2=0x7f4acc150bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc150bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 -- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc158770 msgr2=0x7f4acc151100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc158770 0x7f4acc151100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 -- 172.21.15.70:0/2527009688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc1533f0 con 0x7f4acc072c60 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.732+0000 7f4acb7fe640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0017e0 tx=0x7f4abc00cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.733+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc005ad0 con 0x7f4acc072c60 2024-04-15T15:17:48.740 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.733+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4abc005c90 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.733+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc1536d0 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.733+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4abc0072b0 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.733+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc153c90 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.734+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc0689e0 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.734+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f4abc007450 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.734+0000 7f4ab3fff640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 0x7f4ab806ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.735+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f4abc08b6d0 con 0x7f4acc072c60 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.735+0000 7f4acaffd640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 0x7f4ab806ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:48.741 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.735+0000 7f4acaffd640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 0x7f4ab806ee40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009560 tx=0x7f4ac0009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:48.742 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.739+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f4abc059670 con 0x7f4acc072c60 2024-04-15T15:17:48.917 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.915+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f4acc0b3790 con 0x7f4acc072c60 2024-04-15T15:17:48.920 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.918+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v16)=0 v16) ==== 123+0+0 (secure 0 0 0) 0x7f4abc05d970 con 0x7f4acc072c60 2024-04-15T15:17:48.922 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.919+0000 7f4ab3fff640 1 -- 172.21.15.70:0/2527009688 <== mon.0 v2:172.21.15.70:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f4abc089e50 con 0x7f4acc072c60 2024-04-15T15:17:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 msgr2=0x7f4ab806ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 0x7f4ab806ee40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009560 tx=0x7f4ac0009190 comp rx=0 tx=0).stop 2024-04-15T15:17:48.923 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 msgr2=0x7f4acc0b0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f4abc0017e0 tx=0x7f4abc00cbe0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 shutdown_connections 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f4acc158770 0x7f4acc151100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f4acc0b5230 0x7f4acc150bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f4ab806c980 0x7f4ab806ee40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 --2- 172.21.15.70:0/2527009688 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4acc072c60 0x7f4acc0b0a50 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 >> 172.21.15.70:0/2527009688 conn(0x7f4acc06be00 msgr2=0x7f4acc1593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:48.924 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 shutdown_connections 2024-04-15T15:17:48.925 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:48.921+0000 7f4ad1ace640 1 -- 172.21.15.70:0/2527009688 wait complete. 2024-04-15T15:17:48.934 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-15T15:17:48.935 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-15T15:17:49.037 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.035+0000 7f3970bc8640 1 Processor -- start 2024-04-15T15:17:49.038 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.035+0000 7f3970bc8640 1 -- start start 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.036+0000 7f3970bc8640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f396c0d3270 0x7f396c0cf370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.036+0000 7f3970bc8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f396c0b47a0 con 0x7f396c15e700 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.036+0000 7f3970bc8640 1 -- --> v1:172.21.15.96:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f396c0b4a80 con 0x7f396c0d2360 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.036+0000 7f3970bc8640 1 -- --> v2:172.21.15.70:3300/0 -- mon_getmap magic: 0 -- 0x7f396c0b53b0 con 0x7f396c0d3270 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396ad9d640 1 --2- >> v2:172.21.15.70:3300/0 conn(0x7f396c0d3270 0x7f396c0cf370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f396c15e700 0x7f396c0b5840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:44288/0 (socket says 172.21.15.70:44288) 2024-04-15T15:17:49.039 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396a59c640 1 -- 172.21.15.70:0/2204012582 learned_addr learned my addr 172.21.15.70:0/2204012582 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-04-15T15:17:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396ad9d640 1 -- 172.21.15.70:0/2204012582 >> v1:172.21.15.96:6789/0 conn(0x7f396c0d2360 legacy=0x7f396c0d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-15T15:17:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396ad9d640 1 -- 172.21.15.70:0/2204012582 >> v1:172.21.15.70:6789/0 conn(0x7f396c15e700 legacy=0x7f396c0b5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396ad9d640 1 -- 172.21.15.70:0/2204012582 --> v2:172.21.15.70:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c0b4d60 con 0x7f396c0d3270 2024-04-15T15:17:49.040 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.037+0000 7f396959a640 1 -- 172.21.15.70:0/2204012582 <== mon.0 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 841176395 0 0) 0x7f396c0b47a0 con 0x7f396c15e700 2024-04-15T15:17:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f396ad9d640 1 --2- 172.21.15.70:0/2204012582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c0cf370 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f396c0b5530 tx=0x7f396402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccab2a3fc01c1748 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.041 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f396959a640 1 -- 172.21.15.70:0/2204012582 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964035020 con 0x7f396c0d3270 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f396959a640 1 -- 172.21.15.70:0/2204012582 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f3964031830 con 0x7f396c0d3270 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f396959a640 1 -- 172.21.15.70:0/2204012582 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3964031c20 con 0x7f396c0d3270 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 -- 172.21.15.70:0/2204012582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 msgr2=0x7f396c0cf370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 --2- 172.21.15.70:0/2204012582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c0cf370 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f396c0b5530 tx=0x7f396402b8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 -- 172.21.15.70:0/2204012582 shutdown_connections 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 --2- 172.21.15.70:0/2204012582 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c0cf370 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 -- 172.21.15.70:0/2204012582 >> 172.21.15.70:0/2204012582 conn(0x7f396c08bf60 msgr2=0x7f396c0d4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:49.042 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 -- 172.21.15.70:0/2204012582 shutdown_connections 2024-04-15T15:17:49.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.038+0000 7f3970bc8640 1 -- 172.21.15.70:0/2204012582 wait complete. 2024-04-15T15:17:49.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 Processor -- start 2024-04-15T15:17:49.043 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 -- start start 2024-04-15T15:17:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 --2- >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f396c0d2360 0x7f396c0b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.045 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 --2- >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f396c15e700 0x7f396c0b9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0d1d40 con 0x7f396c0d3270 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 -- --> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] -- mon_getmap magic: 0 -- 0x7f396c0b4450 con 0x7f396c15e700 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3970bc8640 1 -- --> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] -- mon_getmap magic: 0 -- 0x7f396c0d1ec0 con 0x7f396c0d2360 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3969d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3969d9b640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:53806/0 (socket says 172.21.15.70:53806) 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.040+0000 7f3969d9b640 1 -- 172.21.15.70:0/3506425157 learned_addr learned my addr 172.21.15.70:0/3506425157 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f396ad9d640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f396c15e700 0x7f396c0b9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.046 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f396a59c640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f396c0d2360 0x7f396c0b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 -- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f396c0d2360 msgr2=0x7f396c0b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f396c0d2360 0x7f396c0b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 -- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f396c15e700 msgr2=0x7f396c0b9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f396c15e700 0x7f396c0b9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 -- 172.21.15.70:0/3506425157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f396c15d7e0 con 0x7f396c0d3270 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3969d9b640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f396000c410 tx=0x7f396000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960017070 con 0x7f396c0d3270 2024-04-15T15:17:49.047 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f39600035f0 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f396c0b8170 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3960003a40 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.041+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f396c0b8730 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.042+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f396c0689e0 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.043+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 6) ==== 91091+0+0 (secure 0 0 0) 0x7f3960003be0 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.043+0000 7f395affd640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 0x7f394c06ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.043+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5421+0+0 (secure 0 0 0) 0x7f396008aeb0 con 0x7f396c0d3270 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.043+0000 7f396a59c640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 0x7f394c06ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-15T15:17:49.048 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.043+0000 7f396a59c640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 0x7f394c06ee70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f395c007fd0 tx=0x7f395c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-15T15:17:49.049 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.047+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187246 (secure 0 0 0) 0x7f3960059300 con 0x7f396c0d3270 2024-04-15T15:17:49.264 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.262+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f396c0ba500 con 0x7f396c0d3270 2024-04-15T15:17:49.265 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.263+0000 7f395affd640 1 -- 172.21.15.70:0/3506425157 <== mon.0 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+3866 (secure 0 0 0) 0x7f39600010c0 con 0x7f396c0d3270 2024-04-15T15:17:49.265 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-04-15T15:17:49.266 INFO:teuthology.orchestra.run.smithi070.stdout:{"epoch":22,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4895,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6836/137170453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":137170453},{"type":"v1","addr":"172.21.15.70:6836","nonce":137170453}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":4898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/2694985052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6835","nonce":2694985052},{"type":"v1","addr":"172.21.15.70:6837","nonce":2694985052}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":4904,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6834/1995327138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6832","nonce":1995327138},{"type":"v1","addr":"172.21.15.96:6834","nonce":1995327138}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5689,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.96:6835/3912559408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6833","nonce":3912559408},{"type":"v1","addr":"172.21.15.96:6835","nonce":3912559408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":22}],"filesystems":[]} 2024-04-15T15:17:49.266 INFO:teuthology.orchestra.run.smithi070.stderr:dumped fsmap epoch 22 2024-04-15T15:17:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 msgr2=0x7f394c06ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 0x7f394c06ee70 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f395c007fd0 tx=0x7f395c009040 comp rx=0 tx=0).stop 2024-04-15T15:17:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 msgr2=0x7f396c15d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-15T15:17:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f396000c410 tx=0x7f396000c8d0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.267 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 shutdown_connections 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] conn(0x7f396c15e700 0x7f396c0b9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:6832/2172332499,v1:172.21.15.70:6833/2172332499] conn(0x7f394c06c9b0 0x7f394c06ee70 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f396c0d3270 0x7f396c15d2a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 --2- 172.21.15.70:0/3506425157 >> [v2:172.21.15.96:3301/0,v1:172.21.15.96:6790/0] conn(0x7f396c0d2360 0x7f396c0b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 >> 172.21.15.70:0/3506425157 conn(0x7f396c08bf60 msgr2=0x7f396c0b5840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-15T15:17:49.268 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 shutdown_connections 2024-04-15T15:17:49.269 INFO:teuthology.orchestra.run.smithi070.stderr:2024-04-15T15:17:49.265+0000 7f3970bc8640 1 -- 172.21.15.70:0/3506425157 wait complete. 2024-04-15T15:17:49.278 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-15T15:17:49.279 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-15T15:17:49.279 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-15T15:17:49.279 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.288 INFO:tasks.ceph.mds.a:Stopped 2024-04-15T15:17:49.288 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-15T15:17:49.289 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.338 INFO:tasks.ceph.mds.c:Stopped 2024-04-15T15:17:49.338 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-15T15:17:49.338 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.350 INFO:tasks.ceph.mds.b:Stopped 2024-04-15T15:17:49.350 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-15T15:17:49.350 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.402 INFO:tasks.ceph.mds.d:Stopped 2024-04-15T15:17:49.403 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-15T15:17:49.403 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-15T15:17:49.403 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.433 INFO:tasks.ceph.osd.0:Stopped 2024-04-15T15:17:49.433 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-15T15:17:49.433 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.507 INFO:tasks.ceph.osd.1:Stopped 2024-04-15T15:17:49.508 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-15T15:17:49.508 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.583 INFO:tasks.ceph.osd.2:Stopped 2024-04-15T15:17:49.583 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-15T15:17:49.583 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.658 INFO:tasks.ceph.osd.3:Stopped 2024-04-15T15:17:49.658 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-15T15:17:49.659 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.693 INFO:tasks.ceph.osd.4:Stopped 2024-04-15T15:17:49.694 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-15T15:17:49.694 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.762 INFO:tasks.ceph.osd.5:Stopped 2024-04-15T15:17:49.763 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-15T15:17:49.763 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.840 INFO:tasks.ceph.osd.6:Stopped 2024-04-15T15:17:49.841 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-15T15:17:49.841 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.911 INFO:tasks.ceph.osd.7:Stopped 2024-04-15T15:17:49.912 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-15T15:17:49.912 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-15T15:17:49.912 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.939 INFO:tasks.ceph.mgr.y:Stopped 2024-04-15T15:17:49.939 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-15T15:17:49.939 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.979 INFO:tasks.ceph.mgr.x:Stopped 2024-04-15T15:17:49.980 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-15T15:17:49.980 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-15T15:17:49.980 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:49.991 INFO:tasks.ceph.mon.a:Stopped 2024-04-15T15:17:49.992 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-15T15:17:49.992 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:50.032 INFO:tasks.ceph.mon.b:Stopped 2024-04-15T15:17:50.033 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-15T15:17:50.033 INFO:teuthology.orchestra.run:waiting for 300 2024-04-15T15:17:50.085 INFO:tasks.ceph.mon.c:Stopped 2024-04-15T15:17:50.085 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-15T15:17:50.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | head -n 1 2024-04-15T15:17:50.121 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:50.121 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-15T15:17:50.342 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:50.342 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-15T15:17:50.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:50.424 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-15T15:17:50.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:50.481 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-15T15:17:50.540 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:50.540 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-15T15:17:51.014 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:51.014 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-15T15:17:51.087 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:51.087 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-15T15:17:51.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:51.143 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-15T15:17:51.211 INFO:tasks.ceph:Archiving mon data... 2024-04-15T15:17:51.212 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/data/mon.a.tgz 2024-04-15T15:17:51.212 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-04-15T15:17:51.228 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.7AUElOKFhb 2024-04-15T15:17:51.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7AUElOKFhb 2024-04-15T15:17:51.373 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.7AUElOKFhb 2024-04-15T15:17:51.459 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.7AUElOKFhb is 357KB 2024-04-15T15:17:51.555 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.7AUElOKFhb 2024-04-15T15:17:51.571 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/data/mon.b.tgz 2024-04-15T15:17:51.571 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-04-15T15:17:51.589 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.E60qWyrS1r 2024-04-15T15:17:51.589 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.E60qWyrS1r 2024-04-15T15:17:51.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.E60qWyrS1r 2024-04-15T15:17:51.843 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.E60qWyrS1r is 478KB 2024-04-15T15:17:51.913 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.E60qWyrS1r 2024-04-15T15:17:51.929 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/data/mon.c.tgz 2024-04-15T15:17:51.929 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-04-15T15:17:51.983 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.8QlQxtlVUx 2024-04-15T15:17:51.984 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.8QlQxtlVUx 2024-04-15T15:17:52.137 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.8QlQxtlVUx 2024-04-15T15:17:52.224 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.8QlQxtlVUx is 478KB 2024-04-15T15:17:52.322 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.8QlQxtlVUx 2024-04-15T15:17:52.338 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-15T15:17:52.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:17:52.341 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-15T15:17:52.429 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:17:52.460 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-15T15:17:52.461 INFO:tasks.ceph:Archiving crash dumps... 2024-04-15T15:17:52.463 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi070/crash 2024-04-15T15:17:52.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:17:52.500 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi096/crash 2024-04-15T15:17:52.501 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-15T15:17:52.528 INFO:tasks.ceph:Shutting down logrotate 2024-04-15T15:17:52.529 DEBUG:teuthology.orchestra.run.smithi070:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:52.544 DEBUG:teuthology.orchestra.run.smithi096:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:52.613 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:52.644 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-15T15:17:52.681 INFO:tasks.ceph:Compressing logs... 2024-04-15T15:17:52.682 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:17:52.686 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T15:17:52.714 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43110.log 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr:gzip/var/log/ceph/ceph-client.admin.43110.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.54873.log 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44339.log 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43110.log.gz 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2024-04-15T15:17:52.715 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42611.log 2024-04-15T15:17:52.717 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54873.log.gz 2024-04-15T15:17:52.717 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49056.log 2024-04-15T15:17:52.717 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44339.log.gz 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43278.log 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49875.log 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42611.log.gz 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49056.log.gz 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56262.log 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43278.log: /var/log/ceph/ceph-client.admin.49875.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.43278.log.gz --verbose 2024-04-15T15:17:52.718 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.52904.log 2024-04-15T15:17:52.774 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-15T15:17:52.774 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37795.log 2024-04-15T15:17:52.774 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.45996.log 2024-04-15T15:17:52.774 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.37795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37554.log 2024-04-15T15:17:52.774 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.3.45996.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37795.log.gz 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43988.log 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37554.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.1.43988.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.37554.log.gz 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45162.log 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.3.45996.log.gz 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.1.43988.log.gz 2024-04-15T15:17:52.775 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.1.45162.log: /var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 91.2% --verbose -- /var/log/ceph/ceph-client.admin.37635.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.1.45162.log.gz 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37675.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37715.log 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr: 88.4%/var/log/ceph/ceph-client.admin.37675.log: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:17:52.776 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37635.log.gz 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37675.log.gz 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44444.log 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37715.log.gz 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37755.log 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:gzip/var/log/ceph/ceph-client.3.44444.log: -5 --verbose -- /var/log/ceph/ceph-client.1.45656.log 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37755.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37755.log.gz 2024-04-15T15:17:52.777 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.1.45656.log: 91.3% -- replaced with /var/log/ceph/ceph-client.3.44444.log.gz 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mon.b.log: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.1.45656.log.gz 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37514.log 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37594.log 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37514.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37514.log.gz 2024-04-15T15:17:52.778 INFO:teuthology.orchestra.run.smithi096.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-15T15:17:52.779 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-15T15:17:52.779 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.37594.log: /var/log/ceph/ceph-mon.c.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.37594.log.gz 2024-04-15T15:17:52.835 INFO:teuthology.orchestra.run.smithi096.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56262.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49875.log.gz 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42867.log 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56262.log.gz 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52904.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49503.log 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52904.log.gz 2024-04-15T15:17:52.860 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42867.log.gz 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46349.log 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57311.log 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46349.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49503.log.gz 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46139.log 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46349.log.gz 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46265.log 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57311.log.gz 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58683.log 2024-04-15T15:17:52.861 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46265.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46139.log.gz 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46181.log 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46265.log.gz 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56491.log 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58683.log.gz 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56737.log 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56491.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46181.log.gz 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50967.log 2024-04-15T15:17:52.862 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8%/var/log/ceph/ceph-client.admin.56737.log: -- replaced with /var/log/ceph/ceph-client.admin.56491.log.gz 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44381.log 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56737.log.gz 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53156.log 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44381.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50967.log.gz 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44381.log.gz 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48123.log 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53156.log.gz 2024-04-15T15:17:52.863 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44795.log 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48123.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56024.log 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48123.log.gz 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48258.log 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.44795.log.gz 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56695.log 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44963.log 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48258.log.gz 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56695.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.56024.log.gz 2024-04-15T15:17:52.864 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54215.log 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56695.log.gz 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44255.log 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44963.log.gz 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50537.log 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54215.log.gz 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56575.log 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50537.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44255.log.gz 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50453.log 2024-04-15T15:17:52.865 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50537.log.gz 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54621.log 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50453.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56575.log.gz 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50763.log 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6%/var/log/ceph/ceph-client.admin.54621.log: -- replaced with /var/log/ceph/ceph-client.admin.50453.log.gz 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54621.log.gz 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45139.log 2024-04-15T15:17:52.866 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50763.log.gz 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53072.log 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45139.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.2.47744.log 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45139.log.gz 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51573.log 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2%/var/log/ceph/ceph-client.2.47744.log: -- replaced with /var/log/ceph/ceph-client.admin.53072.log.gz 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53891.log 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57437.log 2024-04-15T15:17:52.867 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51573.log.gz 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53891.log: 91.0% -- replaced with /var/log/ceph/ceph-client.2.47744.log.gz 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49629.log 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53891.log.gz 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49445.log 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57437.log.gz 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46223.log 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56533.log 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49629.log.gz 2024-04-15T15:17:52.868 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46223.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49445.log.gz 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56653.log 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46223.log.gz 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48036.log 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56653.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.49713.log 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.56533.log.gz 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48036.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56653.log.gz 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48036.log.gz 2024-04-15T15:17:52.869 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8%/var/log/ceph/ceph-client.admin.50679.log: -- replaced with /var/log/ceph/ceph-client.admin.49713.log.gz 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52564.log 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42737.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50721.log 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45145.log 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52564.log.gz 2024-04-15T15:17:52.870 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51867.log 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.45145.log: -- replaced with /var/log/ceph/ceph-client.admin.50721.log.gz 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56176.log 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45145.log.gz 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49140.log 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51867.log.gz 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54089.log 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49140.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56176.log.gz -5 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2024-04-15T15:17:52.871 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54089.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49140.log.gz 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58515.log 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54089.log.gz 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51471.log 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56963.log 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51471.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58515.log.gz 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51657.log 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51471.log.gz 2024-04-15T15:17:52.872 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50327.log 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56963.log.gz 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49403.log 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2%/var/log/ceph/ceph-client.admin.50327.log: -- replaced with /var/log/ceph/ceph-client.admin.51657.log.gz 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45136.log 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49403.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50327.log.gz 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54315.log 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49403.log.gz 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57941.log 2024-04-15T15:17:52.873 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54315.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45136.log.gz 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46013.log 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57941.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54315.log.gz 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48342.log 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57941.log.gz 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50043.log 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46013.log.gz 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45466.log 2024-04-15T15:17:52.874 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50043.log: gzip -5 88.9% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.48342.log.gz /var/log/ceph/ceph-client.admin.44071.log 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3%/var/log/ceph/ceph-client.admin.45466.log: -- replaced with /var/log/ceph/ceph-client.admin.50043.log.gz 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44071.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45466.log.gz 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45480.log 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44071.log.gz 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37694.log 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2024-04-15T15:17:52.875 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44423.log 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45480.log.gz 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46055.log 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37694.log.gz 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58599.log 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44423.log.gz 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58347.log 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46055.log.gz 2024-04-15T15:17:52.876 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57167.log 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58347.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58599.log.gz 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50883.log 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58347.log.gz 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52736.log 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50883.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57167.log.gz 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49833.log 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50883.log.gz 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59121.log 2024-04-15T15:17:52.877 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52736.log.gz 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42695.log 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.59121.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49833.log.gz 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42964.log 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.42695.log: -- replaced with /var/log/ceph/ceph-client.admin.59121.log.gz 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58893.log 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42695.log.gz 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48930.log 2024-04-15T15:17:52.878 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58893.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42964.log.gz 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42918.log 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48930.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58893.log.gz 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50925.log 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48930.log.gz 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51741.log 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50925.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42918.log.gz 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43320.log 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51741.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50925.log.gz 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58025.log 2024-04-15T15:17:52.879 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51741.log.gz 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37797.log 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48846.log 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37797.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.43320.log.gz 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58025.log.gz 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58725.log 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37797.log.gz 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52269.log 2024-04-15T15:17:52.880 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48846.log: /var/log/ceph/ceph-client.admin.58725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54789.log 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48846.log.gz 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52269.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58725.log.gz 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50495.log 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54789.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52269.log.gz 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43236.log 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54789.log.gz 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52396.log 2024-04-15T15:17:52.881 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50495.log.gz 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52522.log 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45812.log 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43236.log.gz 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52522.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52396.log.gz 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43987.log 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52522.log.gz 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54257.log 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43987.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45812.log.gz 2024-04-15T15:17:52.882 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49959.log 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43987.log.gz 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56407.log 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54257.log.gz 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54501.log 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56407.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49959.log.gz 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57689.log 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56407.log.gz 2024-04-15T15:17:52.883 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54501.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54501.log.gz 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54357.log 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57689.log.gz 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45142.log 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54357.log.gz 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46307.log 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2024-04-15T15:17:52.884 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54831.log 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49361.log 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.45142.log.gz 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54831.log: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46307.log.gz 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45479.log 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54831.log.gz 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45813.log 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49277.log 2024-04-15T15:17:52.885 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45479.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49361.log.gz 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58067.log 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45479.log.gz 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49277.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45813.log.gz 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58263.log 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.58067.log: -- replaced with /var/log/ceph/ceph-client.admin.49277.log.gz 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58935.log 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58263.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58067.log.gz 2024-04-15T15:17:52.886 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52692.log 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58263.log.gz 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51177.log 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58935.log.gz 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53975.log 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51177.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.52692.log.gz --verbose 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr: -- /var/log/ceph/ceph-client.admin.46685.log 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51177.log.gz 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50127.log 2024-04-15T15:17:52.887 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1%/var/log/ceph/ceph-client.admin.46685.log: -- replaced with /var/log/ceph/ceph-client.admin.53975.log.gz 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59019.log 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7%/var/log/ceph/ceph-client.admin.50127.log: -- replaced with /var/log/ceph/ceph-client.admin.46685.log.gz 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57047.log 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50127.log.gz 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.59019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57479.log 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57047.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59019.log.gz 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57251.log 2024-04-15T15:17:52.888 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57479.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57047.log.gz 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48636.log 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57479.log.gz 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56821.log 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57251.log.gz 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56879.log 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56821.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.48636.log.gz 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57857.log 2024-04-15T15:17:52.889 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56821.log.gz 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48384.log 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57857.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56879.log.gz 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44297.log 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48384.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.57857.log.gz 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51970.log 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48384.log.gz 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46475.log 2024-04-15T15:17:52.890 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51970.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44297.log.gz 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57899.log 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46475.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51970.log.gz 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48804.log 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46475.log.gz 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51699.log 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57899.log.gz 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54705.log 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51699.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48804.log.gz 2024-04-15T15:17:52.891 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45143.log 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51699.log.gz 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48720.log 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%/var/log/ceph/ceph-client.admin.45143.log: -- replaced with /var/log/ceph/ceph-client.admin.54705.log.gz 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57563.log 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53933.log 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.45143.log.gz 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48720.log.gz 2024-04-15T15:17:52.892 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54957.log 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57563.log.gz 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46391.log 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54957.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53933.log.gz 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54915.log 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3%/var/log/ceph/ceph-client.admin.46391.log: -- replaced with /var/log/ceph/ceph-client.admin.54957.log.gz 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48972.log 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46391.log.gz 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54579.log 2024-04-15T15:17:52.893 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48972.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.54915.log.gz -5 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52862.log 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48972.log.gz/var/log/ceph/ceph-client.admin.54579.log: 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45138.log 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50001.log 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54579.log.gz 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45138.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52862.log.gz 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.45138.log.gz 2024-04-15T15:17:52.894 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57731.log 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52778.log 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50001.log.gz 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58305.log 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52778.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57731.log.gz 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.52778.log.gz 2024-04-15T15:17:52.895 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50369.log 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31413.log 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58305.log.gz 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48300.log 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.tmp-client.admin.31413.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50369.log.gz 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31413.log.gz 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49235.log 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53692.log 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48300.log.gz 2024-04-15T15:17:52.896 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59061.log 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51009.log 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.53692.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49235.log.gz 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.59061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42823.log 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2%/var/log/ceph/ceph-client.admin.51009.log: -- replaced with /var/log/ceph/ceph-client.admin.59061.log.gz 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51783.log 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.0.53692.log.gz 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51009.log.gz 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42349.log 2024-04-15T15:17:52.897 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52988.log 2024-04-15T15:17:52.898 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.42823.log.gz 2024-04-15T15:17:52.898 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51783.log.gz 2024-04-15T15:17:52.898 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52227.log 2024-04-15T15:17:52.898 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50253.log 88.7% 2024-04-15T15:17:52.898 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42349.log.gz 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52988.log: /var/log/ceph/ceph-client.admin.52227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58641.log 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52988.log.gz 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52227.log.gz 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49545.log 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50253.log: /var/log/ceph/ceph-client.admin.58641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53366.log 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50253.log.gz 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49545.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58641.log.gz 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51135.log 2024-04-15T15:17:52.899 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49545.log.gz 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43068.log 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57395.log 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.53366.log.gz 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51135.log.gz 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43068.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56132.log 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49587.log 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43068.log.gz 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.57395.log.gz 2024-04-15T15:17:52.900 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45477.log 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54173.log 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56132.log.gz 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49587.log: /var/log/ceph/ceph-client.admin.45477.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49587.log.gz 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54663.log 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45477.log.gz 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59161.log 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54173.log: /var/log/ceph/ceph-client.admin.54663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51825.log 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54173.log.gz 2024-04-15T15:17:52.901 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54663.log.gz 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.59161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48762.log 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59161.log.gz 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52820.log 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51825.log: /var/log/ceph/ceph-client.admin.48762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45816.log 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51825.log.gz 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52820.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48762.log.gz 2024-04-15T15:17:52.902 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53432.log 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52820.log.gz 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56304.log 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45816.log: /var/log/ceph/ceph-client.admin.53432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48552.log 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45816.log.gz 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56304.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53432.log.gz 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57983.log 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58851.log 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48552.log.gz 2024-04-15T15:17:52.903 INFO:teuthology.orchestra.run.smithi070.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.56304.log.gz 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52055.log 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57983.log: /var/log/ceph/ceph-client.admin.58851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44837.log 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57983.log.gz 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58851.log.gz 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50169.log 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52055.log.gz 2024-04-15T15:17:52.904 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45815.log 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44837.log: /var/log/ceph/ceph-client.admin.50169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58431.log 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44837.log.gz 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50169.log.gz 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46855.log 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45815.log.gz 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58431.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48594.log 2024-04-15T15:17:52.905 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58431.log.gz 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46855.log: /var/log/ceph/ceph-osd.3.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46855.log.gz 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53324.log 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52311.log 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48594.log: /var/log/ceph/ceph-client.admin.53324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44197.log 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48594.log.gz 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53324.log.gz 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46433.log 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52311.log.gz 2024-04-15T15:17:52.906 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46097.log 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42445.log 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44197.log.gz 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46433.log: /var/log/ceph/ceph-client.admin.46097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56090.log 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46433.log.gz 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42445.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46097.log.gz 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45144.log 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42445.log.gz 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57005.log 2024-04-15T15:17:52.907 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52145.log 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5%/var/log/ceph/ceph-client.admin.45144.log: -- replaced with /var/log/ceph/ceph-client.admin.56090.log.gz 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56779.log 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45144.log.gz 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52145.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49098.log 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57005.log.gz 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52145.log.gz 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54747.log 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-04-15T15:17:52.908 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49098.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56779.log.gz 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44113.log 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49098.log.gz 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54747.log.gz 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53282.log 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44113.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48510.log 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44113.log.gz 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46559.log 2024-04-15T15:17:52.909 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48510.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53282.log.gz 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51219.log 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48510.log.gz 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46643.log 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46559.log.gz 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57815.log 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.55717.log 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51219.log.gz/var/log/ceph/ceph-client.admin.46643.log: 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58557.log 2024-04-15T15:17:52.910 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46643.log.gz 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.2.55717.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57815.log.gz 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57209.log 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58557.log.gz 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57209.log: gzip 91.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.2.55717.log.gz -- 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr: /var/log/ceph/ceph-client.admin.45478.log 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42779.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57209.log.gz 2024-04-15T15:17:52.911 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57773.log 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45478.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43796.log 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45478.log.gz 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55457.log 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51615.log 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43796.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57773.log.gz 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48171.log 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55457.log.gz 2024-04-15T15:17:52.912 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44543.log 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5/var/log/ceph/ceph-client.admin.48171.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51615.log.gz 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43026.log 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44543.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48171.log.gz 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45481.log 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44543.log.gz 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51345.log 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43026.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.43796.log.gz 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45481.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43026.log.gz 2024-04-15T15:17:52.913 INFO:teuthology.orchestra.run.smithi070.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54131.log 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45481.log.gz 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44627.log 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51345.log.gz 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50637.log 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54131.log: /var/log/ceph/ceph-client.admin.44627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55415.log 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54131.log.gz 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44627.log.gz/var/log/ceph/ceph-client.admin.50637.log: 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43194.log 2024-04-15T15:17:52.914 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55415.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50637.log.gz 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55415.log.gz 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53240.log 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mgr.y.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43194.log.gz 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47926.log 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56449.log 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53240.log.gz/var/log/ceph/ceph-client.admin.47926.log: 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43152.log 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47926.log.gz 2024-04-15T15:17:52.915 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50085.log 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43152.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56449.log.gz 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58767.log 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.43152.log.gz 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44753.log 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50085.log.gz 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53198.log 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44753.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58767.log.gz 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37428.log 2024-04-15T15:17:52.916 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44753.log.gz 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59203.log 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53198.log.gz 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.37428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53849.log 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.59203.log: gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.43386.log 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37428.log.gz 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59203.log.gz 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45469.log 2024-04-15T15:17:52.917 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53849.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54459.log 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.43386.log: /var/log/ceph/ceph-client.admin.45469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.43386.log.gz 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47866.log 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45469.log.gz 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54459.log: /var/log/ceph/ceph-client.admin.42526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57353.log 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr: 88.8% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47866.log: -- replaced with /var/log/ceph/ceph-client.admin.54459.log.gz 2024-04-15T15:17:52.918 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48678.log 2024-04-15T15:17:52.919 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47866.log.gz 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44711.log 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57353.log.gz 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49014.log 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48678.log.gz 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58977.log 2024-04-15T15:17:52.922 INFO:teuthology.orchestra.run.smithi070.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44711.log.gz 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57605.log 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.49014.log.gz 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr: 87.3%/var/log/ceph/ceph-client.admin.57605.log: -- replaced with /var/log/ceph/ceph-client.admin.58977.log.gz 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56218.log 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57605.log.gz 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.42568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44029.log 2024-04-15T15:17:52.923 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46727.log 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56218.log.gz 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45817.log 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44029.log.gz 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.47246.log 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45817.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.46727.log.gz 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr: 87.4%/var/log/ceph/ceph-client.0.47246.log: -- replaced with /var/log/ceph/ceph-client.admin.45817.log.gz 2024-04-15T15:17:52.924 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48079.log 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55032.log 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.0.log: 90.2% -- replaced with /var/log/ceph/ceph-client.0.47246.log.gz 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47368.log 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58809.log 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48079.log.gz 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.47368.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55032.log.gz 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52438.log 2024-04-15T15:17:52.925 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47368.log.gz/var/log/ceph/ceph-client.admin.58809.log: 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52480.log 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44155.log 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58809.log.gz 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52480.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52438.log.gz 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52480.log.gz 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57125.log 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.44155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46601.log 2024-04-15T15:17:52.926 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51261.log 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44155.log.gz 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57125.log.gz 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46601.log: gzip 89.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.50211.log 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46601.log.gz 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55982.log 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51261.log.gz/var/log/ceph/ceph-client.admin.50211.log: 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53030.log 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50211.log.gz 2024-04-15T15:17:52.927 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51051.log 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55982.log.gz 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53030.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53030.log.gz 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52012.log 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51051.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54399.log 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52012.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51051.log.gz 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57647.log 2024-04-15T15:17:52.928 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52012.log.gz 84.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54399.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51303.log 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54399.log.gz 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.2.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57647.log.gz 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52946.log 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48888.log 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51303.log: /var/log/ceph/ceph-client.admin.52946.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51303.log.gz 2024-04-15T15:17:52.929 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52946.log.gz 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45814.log 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49791.log 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48888.log: /var/log/ceph/ceph-client.admin.45814.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45814.log.gz 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48888.log.gz 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45135.log 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49791.log: /var/log/ceph/ceph-mds.c.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49791.log.gz 2024-04-15T15:17:52.930 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48468.log 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45135.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.45135.log.gz 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53114.log 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48468.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48468.log.gz 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50411.log 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.53114.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53114.log.gz 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58221.log 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50411.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50411.log.gz 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45472.log 2024-04-15T15:17:52.931 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58221.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58221.log.gz 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48426.log 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45472.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45472.log.gz 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55839.log 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48426.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48426.log.gz 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51429.log 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55839.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55839.log.gz 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49917.log 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51429.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51429.log.gz 2024-04-15T15:17:52.932 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57521.log 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49917.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49917.log.gz 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45677.log 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.57521.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57521.log.gz 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48215.log 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45677.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45677.log.gz 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58389.log 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.48215.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48215.log.gz 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49319.log 2024-04-15T15:17:52.933 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58389.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58389.log.gz 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45811.log 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.49319.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49319.log.gz 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52354.log 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.45811.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45811.log.gz 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51531.log 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52354.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52354.log.gz 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55916.log 2024-04-15T15:17:52.934 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51531.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51531.log.gz 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52606.log 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.55916.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55916.log.gz 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46517.log 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.52606.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52606.log.gz 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51387.log 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.46517.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46517.log.gz 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.55292.log 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51387.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51387.log.gz 2024-04-15T15:17:52.935 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51927.log 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.0.55292.log: 91.7% -- replaced with /var/log/ceph/ceph-client.0.55292.log.gz 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.51927.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51927.log.gz 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54031.log 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50595.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.54031.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54031.log.gz 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50805.log 2024-04-15T15:17:52.936 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mds.a.log: 91.3% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58473.log 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.50805.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50805.log.gz 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56921.log 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.58473.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58473.log.gz 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr: 91.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.56921.log: /var/log/ceph/ceph-client.admin.42257.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56921.log.gz 2024-04-15T15:17:52.937 INFO:teuthology.orchestra.run.smithi070.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-04-15T15:17:52.938 INFO:teuthology.orchestra.run.smithi070.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-15T15:17:52.938 INFO:teuthology.orchestra.run.smithi096.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-15T15:17:52.938 INFO:teuthology.orchestra.run.smithi096.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-15T15:17:52.983 INFO:teuthology.orchestra.run.smithi070.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-15T15:17:53.405 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-15T15:17:53.414 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-15T15:17:53.422 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-15T15:17:53.429 INFO:teuthology.orchestra.run.smithi096.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-15T15:17:53.464 INFO:teuthology.orchestra.run.smithi096.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-15T15:17:53.465 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-04-15T15:17:53.466 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m0.727s 2024-04-15T15:17:53.466 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m2.966s 2024-04-15T15:17:53.466 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.124s 2024-04-15T15:17:53.477 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-15T15:17:53.500 INFO:teuthology.orchestra.run.smithi070.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-15T15:17:53.512 INFO:teuthology.orchestra.run.smithi070.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-15T15:17:53.514 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-04-15T15:17:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m0.816s 2024-04-15T15:17:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m3.157s 2024-04-15T15:17:53.515 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.327s 2024-04-15T15:17:53.515 INFO:tasks.ceph:Archiving logs... 2024-04-15T15:17:53.515 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi070/log 2024-04-15T15:17:53.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:17:54.303 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi096/log 2024-04-15T15:17:54.304 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T15:17:54.757 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-15T15:17:54.770 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-15T15:17:54.771 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:17:54.774 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-15T15:17:54.861 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T15:17:54.870 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T15:17:54.871 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:54.874 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T15:17:54.888 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:54.890 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-04-15T15:17:54.893 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:54.893 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-04-15T15:17:54.893 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 2 +1581us[+1582us] +/- 50ms 2024-04-15T15:17:54.894 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1804us[-1804us] +/- 77ms 2024-04-15T15:17:54.894 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 -640us[ -640us] +/- 74ms 2024-04-15T15:17:54.894 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:54.895 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T15:17:54.895 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-04-15T15:17:54.896 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 6 377 3 +1583us[+1592us] +/- 50ms 2024-04-15T15:17:54.896 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1809us[-1809us] +/- 77ms 2024-04-15T15:17:54.896 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 -641us[ -641us] +/- 74ms 2024-04-15T15:17:54.896 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T15:17:54.896 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T15:17:54.906 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T15:17:54.907 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T15:17:54.970 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:17:55.056 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T15:17:55.135 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:17:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:55.169 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-15T15:17:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:55.202 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T15:17:55.213 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T15:17:55.233 INFO:teuthology.task.internal:Duration was 588.046535 seconds 2024-04-15T15:17:55.233 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T15:17:55.245 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T15:17:55.245 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:17:55.248 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T15:17:55.293 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:17:55.295 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T15:17:55.672 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T15:17:55.673 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-04-15T15:17:55.673 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:17:55.694 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-04-15T15:17:55.695 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T15:17:55.722 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T15:17:55.722 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:17:55.737 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T15:17:55.794 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T15:17:55.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:17:55.808 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T15:17:55.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T15:17:55.949 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T15:17:55.950 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:17:55.978 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T15:17:56.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T15:17:56.018 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:56.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:56.045 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-04-15T15:17:56.075 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-04-15T15:17:56.091 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:56.116 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T15:17:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T15:17:56.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T15:17:56.157 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T15:17:56.158 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi070 2024-04-15T15:17:56.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:17:56.196 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121/remote/smithi096 2024-04-15T15:17:56.196 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T15:17:56.233 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T15:17:56.233 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:17:56.236 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T15:17:56.284 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T15:17:56.294 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T15:17:56.295 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T15:17:56.307 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T15:17:56.307 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:17:56.310 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T15:17:56.322 INFO:teuthology.orchestra.run.smithi070.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:17 /home/ubuntu/cephtest 2024-04-15T15:17:56.338 INFO:teuthology.orchestra.run.smithi096.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 15:17 /home/ubuntu/cephtest 2024-04-15T15:17:56.339 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T15:17:56.351 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T15:17:56.395 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T15:17:56.423 DEBUG:teuthology.nuke:shortname: smithi070 2024-04-15T15:17:56.423 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:17:56.447 DEBUG:teuthology.nuke:shortname: smithi096 2024-04-15T15:17:56.447 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T15:17:56.451 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:07.741710', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoB5nFh91vNFMaqMdEZvAO3Q5iV1zSQy++mNWxmIrkasO1wNgVOeZSJqySUWA+WXmqhQI2uCfETN2LpmTmCYXQ='} 2024-04-15T15:17:56.470 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-04-15_14:46:31-fs-wip-lusov-quiesce-xlock-distro-default-smithi/7657121', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 14:59:07.739664', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJEsMQ1pqCEDgo13JQ7ZcpjjN11GlQS6KwaEcdrg7vuceBTRN9SO5GlKQAlCVhLwlf/pgcNfejnvWruRhTvlfgA='} 2024-04-15T15:17:56.503 INFO:teuthology.orchestra.console.smithi070:Power off 2024-04-15T15:17:56.504 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:17:56.521 INFO:teuthology.orchestra.console.smithi096:Power off 2024-04-15T15:17:56.521 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T15:17:56.531 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-04-15T15:17:56.531 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:56.544 DEBUG:teuthology.orchestra.console.smithi096:power off output: Chassis Power Control: Down/Off 2024-04-15T15:17:56.545 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:17:56.555 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-15T15:17:56.570 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-15T15:18:00.557 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:00.571 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:00.668 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-04-15T15:18:00.681 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is on 2024-04-15T15:18:04.672 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:04.682 DEBUG:teuthology.orchestra.console.smithi096:pexpect command: ipmitool -H smithi096.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T15:18:04.782 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-04-15T15:18:04.793 DEBUG:teuthology.orchestra.console.smithi096:check power output: Chassis Power is off 2024-04-15T15:18:04.885 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-04-15T15:18:04.894 INFO:teuthology.orchestra.console.smithi096:Power off completed 2024-04-15T15:18:05.001 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/quiesce} duration: 588.0465354919434 failure_reason: 'Test failure: test_quiesce_blocked (tasks.cephfs.test_quiesce.TestQuiesce)' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e5920f1f213b4d82bc133749b3664e3f status: fail success: false 2024-04-15T15:18:05.001 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T15:18:05.083 INFO:teuthology.run:FAIL